slow peer reconnect/gratuitous disconnects of peers

[GoogleCodeExporter]

Hello.
We use libtorrent-based client application to distribute content to our 
customers.
But when more than 100 clients trying to download from our server, some users 
experience problems with downloading speed. I can see problem from my home, too:
I have 30 Mbit/s internet connection and when our tracker reports more than 100 
leeachers for a specific torrent, when I'm trying to download this torrent, 
download rate (session_status::payload_download_rate) jumps from 100Kb/s to 
1200Kb/s all the time during download till the end. But, when I start download 
another torrent at the same time (via the same application with same settings, 
and from the same server) with no leechers except me at all (or small amount of 
connected peers) - downloading going very well with speed 3 Mb/s. (And also, 
when no one downloads the first torrent, downloading speed is 3 Mb/s, too.)
So, there are plenty of bandwidth at our server, but for unknown reason 
libtorrent doesn't use it (sometimes, it even disconnects from torrent-client 
on the server).
Our configuration details:
- Dedicated Debian-server with 1Gb/s outgoing bandwidth limit.
- Simple seeding client (on the server-side) based on libtorrent 0.15.9 and 
with high_performance_seed() settings.
- nginx as a web server for url seeding (single url_seed entry in each torrent).
- Downloading client app for Windows based on libtorrent 0.16 (almost latest 
version from trunk) with default settings.
I also tried to tune some settings:
    settings.min_announce_interval = 5;
    settings.unchoke_slots_limit = -1;
    settings.allow_multiple_connections_per_ip = true;
    settings.connection_speed = 20;
    settings.connections_limit = 1000;
    settings.listen_queue_size = 100;
    settings.dht_upload_rate_limit = 10000;
With this changes to the default settings, an average download rate raised by 
1.5 times, but payload_download_rate spread increased to 50..2000 Kb/s.
Also, our customers complain about total clogging their internet channel even 
with average download speed 2-3x times less then their internet bandwidth 
limit, but I didn't see that problem, i.e. the win7 system monitor reports 
roughly the same download traffic as payload_download_rate returned by 
libtorrent.
And also, I tried to use another torrent-client to download this torrent 
(specifically, uTorrent 3.0), and download speed also jumps with wide spread! 
But with some time, it becomes much more stable, and uTorrent become 
downloading primarily from seeding client on the our server (with speed 1-2 
Mb/s, may be it even open multiple connections to this single peer on our 
server).
I think the problem may be in non-configured port forwarding for downloading 
client on our customers' machines, so I didn't expect that they give 
significant contribution for outgoing traffic sharing. But I expect, that if 
our server has enough outgoing bandwidth, downloading should go well, 
regardless of configuration of other peers.

Original issue reported on code.google.com by alextret...@gmail.com on 24 Mar 2012 at 3:03

[GoogleCodeExporter]

At first this sounds like you're disk bound on the server. i.e. when too many 
people are requesting bytes faster than what the disk can provide *at random 
access*, there not everyones connection will be saturated.

However, you say you download a different torrent, with fewer peers, from the 
same seed, and that torrent is fast?

Also, is this an upload speed issue or a download speed issue? Do you know if 
the problem is on the server (i.e. your bittorrent seed) or on the downloader 
end? The title suggests that you think the problem is on the downloading client.

If you think it's on the uploading end, I would recommend at least trying trunk 
as the seed, and see how it does.

Are you seeding both with a bittorrent peer and with a web server? When you 
experience this slow down in download rate, do you know if it's primarily 
attributed to the web server or the bittorrent seed? If the problem is on the 
downloading end, I imagine it could be either.

The second issue about "clogging" internet connections, what is the definition 
of that, if it isn't filling the pipe. Filling a packets-per-second quota?

In either case, in order to trouble shoot this further, you should enable stats 
logging on the end where you think the problem is caused (either downloader or 
seeder, or both). You can enable this by adding statistics=on on the bjam 
command line (or add -DTORRENT_STATS when building). If you send me the logs I 
can take a look.

Original comment by ar...@rasterbar.com on 25 Mar 2012 at 6:41

[GoogleCodeExporter]

> At first this sounds like you're disk bound on the server.
I think, this is not in my case, because torrent files are relatively small 
(1.5 Gb) and server has 24 Gb of RAM. And also, at the seeding client on the 
server I've set the following settings:
        session_settings settings(high_performance_seed());
        settings.seed_time_limit = 365 * 24 * 60 * 60;
        settings.share_ratio_limit = 1e10f;
        settings.seed_time_ratio_limit = 1e10f;
        settings.cache_size = 3 * 65536;//3 Gb
        ses.set_settings(settings);
        ses.set_max_connections(3000);
        ses.set_max_uploads(3000);

> I would recommend at least trying trunk as the seed, and see how it does.
Unfortunately, I could not make to work the version from trunk - it crashes at 
startup.
But I'm sure it is not server-side problem, because as I said, when I 
downloading any other torrent (with a little peers count) from that server - 
downloading going fast. And when I start download torrent with many (> 100-150) 
peers (right after successful downloading previous torrent), I can see that 
problem with unstable download rate.

> The second issue about "clogging" internet connections, what is the 
definition of that, if it isn't filling the pipe. Filling a packets-per-second 
quota?
I don't know exactly, because I can not reproduce it, but our customers 
complain about poor performance in other applications using internet connection 
while our downloading client works, despite that it even doesn't download at 
full rate.

> In either case, in order to trouble shoot this further, you should enable 
stats logging
Unfortunately, we have many peers (100-150) not all the time, and now it is 
just about 50 peers and there is no problem and there is nothing to logging. 
And the next peek can occur through a week, or even month.

Actually, I think the problem is in behavior of downloading client - it stop 
seeding right after completion of download (specifically, it destroys the 
session object), because seeding can negatively affect the performance of the 
application itself.
So, fastest peers (with wide bandwidth) complete download very soon (about 
10-15 minutes) and after that stop seeding. And download rate slow down at the 
moments of disconnecting of that peers.
So, may be I should somehow notify other peers at the moment of stop seeding?
Also, I will try to not stop seeding so soon, to be sure is this causing 
problem or not.

Original comment by alextret...@gmail.com on 26 Mar 2012 at 6:52

[GoogleCodeExporter]

Would you mind telling me where it crashes on startup? What the error is? trunk 
is about to be released as stable, so I would like to fix that.

There's really no way of knowing what's going on without the stats from trunk.

Do you get any suspicious alerts? any performance alerts?

Did you try downloading these torrents with other bittorrent clients? If other 
clients have the same behavior, it's likely a property of the swarm or the 
seeds.

Original comment by arvid.no...@gmail.com on 30 Mar 2012 at 2:40

[GoogleCodeExporter]

> Would you mind telling me where it crashes on startup?
Hm. I've just get and tried to build latest version from trunk and it works 
fine.
So now I've moved to the trunk version.
Also I fixed downloading client to not stop seeding right after download and it 
looks like downloading became work better.
If the problem appears again, then I will try to enable stats logging.

Original comment by alextret...@gmail.com on 30 Mar 2012 at 9:25

[GoogleCodeExporter]

Original comment by arvid.no...@gmail.com on 1 Apr 2012 at 12:57

• Changed state: WontFix

[GoogleCodeExporter]

Hi.
Here is the logs on the downloader side. The first (5252.log) was done when 
downloading torrent with many peers (uTorrent show 9 seeds, 77 peers) and it 
downloading was very-very slow - I stopped downloading after 20 minutes with 
just 159 Mb downloaded (of 1812 Mb), i.e. with average speed about 130 Kb/s. 
The second (1412.log) - for torrent with almost no peers. I started it right 
after stopping the first one, and it was completely downloaded in just 9 
minutes! (i.e. at 3.3 Mb/s, about 25 times faster than previous!)
Both torrents has the same size (as they are just different versions of our 
application - latest and one of the previous) and they were downloaded from the 
same server (with one seeding client, and with nginx web-server for url-seed), 
where they are located even in the same folder near each other. And seeding 
client on the server has no limits or any distinct settings for one torrent or 
another.

Original comment by alextret...@gmail.com on 23 Jun 2012 at 11:49

Attachments:

• session_stats.zip

[GoogleCodeExporter]

Thanks for those logs. Did you try running them through parse_session_stats.py?

From what I can tell, in the case where you have a fast download speed, you 
have two fast peers that periodically disconnect from you (or you disconnect 
from them). When there are no other peers around, libtorrent will quickly 
reconnect and be downloading from them again. It looks like when there are more 
peers, they may dilute  the peer list and make it take a longer time to 
reconnect to them, or maybe preventing them from being reconnected. It makes 
sense for the web seed to disconnect regularly. It's normal to limit the number 
of requests per connection.

The question is why libtorrent isn't reconnecting the web seed immediately. 
It's also strange that the second peer (which I take is a local bittorrent 
peer) is disconnected at all, and why it's not reconnected in the slow case.

This is obviously something that should be fixed in libtorrent. I would greatly 
appreciate it if you would help me narrow down exactly what's going on.

What are the error messages you get from libtorrent when these peers 
disconnect? if it's non-descriptive on the download side (i.e. eof, connection 
reset) do you have access to the error messages from the other peers? From what 
I can tell, it looks like a bit over half of those disconnects are connection 
reset. I'm not sure why.

What is your connection speed set to? The default is 20, which should let you 
reconnect just within a few seconds, but for some reason that doesn't seem to 
be happening right.

I could add some more stats logging for web seeds which might reveal some of 
these issues.

Original comment by arvid.no...@gmail.com on 24 Jun 2012 at 9:55

• Changed title: slow peer reconnect/gratuitous disconnects of peers
• Changed state: Accepted

[GoogleCodeExporter]

In the run where you get low download rate, does libtorrent connect to the web 
seed and your bittorrent seed at all?

Original comment by arvid.no...@gmail.com on 25 Jun 2012 at 4:37

[GoogleCodeExporter]

> Did you try running them through parse_session_stats.py?
Unfortunately, this script doesn't generate png-images referred in the html 
file, but just gnuplot-files (I tried to run it under Windows and Debian). 
Still I can manually process gnuplot-files to obtain png's, but anyway I don't 
clearly understand how I can use those plots :)

> What is your connection speed set to? The default is 20
Yes, I left it by default.

> In the run where you get low download rate, does libtorrent connect to the 
web seed and your bittorrent seed at all?
I can't reproduce that test for now (may be at this weekend), but testing now I 
can see that torrent seeding peer (on the server) doesn't appear at the peer 
list (obtained via get_peer_info) in case of downloading low-rate-torrent, and 
always present while downloading fast-rate-torrent.
But web-seed (on the same server) always present in both (and I suppose that it 
was too in that test I've made 2 days ago, still I can't say for sure).

Original comment by alextret...@gmail.com on 25 Jun 2012 at 6:06

[GoogleCodeExporter]

> What is your connection speed set to? The default is 20
Btw, the default is 6, not 20. But I also tried a much higher values (and also 
higher connections_limit than default) in 10 times or more - it helps, but not 
as much, and behave much worse on low speed internet connections.

> What are the error messages you get from libtorrent when these peers 
disconnect?
How can I get those error messages?
When I called set_alert_mask(alert::peer_notification) and logged all alerts, I 
didn't found any references to ip-address of the server seeding peer.

Original comment by alextret...@gmail.com on 26 Jun 2012 at 8:02

[GoogleCodeExporter]

I've tried set_alert_mask(alert::all_categories) and now I found some 
references to seeding client. There are repetitive pair of alerts regarding it:
00:00:03.001 peer_connect_alert: _0.5.10.4 peer (188.127.248.195, Unknown) 
connecting to peer
00:00:05.008 peer_disconnected_alert: _0.5.10.4 peer (188.127.248.195, Unknown) 
disconnecting: End of file

Original comment by alextret...@gmail.com on 26 Jun 2012 at 8:22

[GoogleCodeExporter]

Hm. Perhaps those "End of file" alerts was because of antivirus/firewall on my 
work computer, since previous slow rate run was at my home computer. Now at 
home, I tried to reproduce that test, but I didn't get slow speed, still the 
differences in downloading those 2 torrents are remain.
Here are alerts.log I've just get (alerts_fast.log for 2 seed torrent, and 
alerts_slow.log for 46 seed torrent). Peers list doesn't store in this logs, 
but it was as I already said: i.e. in fast-rate torrent there are both seeding 
bittorrent-client on the server and a web seed, while in low-rate torrent only 
web seed.
As you can see in *_grep.log, there are much time between peer_connect_alert 
and peer_disconnected_alert in one log and opposite in the another log.
As I can see, seeding client on the server is closing connection while 
attempting to download torrent with many peers.
Is there are any per torrent limits on number of connected peers except than 
connections_limit which I left by default (i.e. = 0)?

Original comment by alextret...@gmail.com on 30 Jun 2012 at 10:19

Attachments:

• alerts.zip

[GoogleCodeExporter]

thanks, I'll take a look.

> Is there are any per torrent limits on number of connected peers except than 
connections_limit which I left by default (i.e. = 0)?

yes, there's also a global limit on the session, which defaults to 200.

Original comment by arvid.no...@gmail.com on 30 Jun 2012 at 2:02

[GoogleCodeExporter]

> yes, there's also a global limit on the session, which defaults to 200.
But on the server bt-client I'm using high_performance_seed() settings (where 
this parameter sets to 8000), so this shouldn't be an issue.

Original comment by alextret...@gmail.com on 30 Jun 2012 at 4:14

[GoogleCodeExporter]

Hi!
We are having similar problems:
With few peers, the download is fast (reaches bandwidth limit on client side).
With many peers, the download speed is fluctuating. And I guess that the reason 
is, that the web seed is not used all the time.

For example, I made a test where I was connected to around 100 peers. There 
were phases where the download speed was around 1200kB/s (bandwidth limit). And 
then there were phases (which were between a few seconds and up to 90s) where 
the download speed was around 40kB/s.
Downloading directly from the web seed always reaches the bandwidth limit for 
the whole duration of the download, so it is not a problem of the web seed or 
the network. It seems that libtorrent simply doesn't decide to download from 
the web seed.

Could it be something like: there are no more connection slots available and 
therefore, libtorrent waits until a peer disconnect before opening a connection 
to the web seed?

Our settings are:
settings.auto_upload_slots = false;
settings.choking_algorithm = libtorrent::session_settings::fixed_slots_choker;
settings.allowed_fast_set_size = 0;
settings.mixed_mode_algorithm = libtorrent::session_settings::prefer_tcp;
settings.unchoke_slots_limit = -1;
settings.ignore_limits_on_local_network = false;
settings.download_rate_limit = settings.local_download_rate_limit = 0;
settings.upload_rate_limit = settings.local_upload_rate_limit = 0;
settings.connections_limit = 250;

Is there any solution in sight?

Original comment by jochen.u...@gameforge.com on 12 Feb 2014 at 11:28

[GoogleCodeExporter]

could you enable verbose logging, zip up the log and send it to me at 
arvid@rasterbar.com?

Original comment by arvid.no...@gmail.com on 13 Feb 2014 at 3:15

[GoogleCodeExporter]

Setting high_performance_seed() solved a very similar problem for us with the 
latest libtorrent version.

Original comment by baphomet...@gmail.com on 23 Apr 2014 at 12:08