search

frostalf / libtorrent

Automatically exported from code.google.com/p/libtorrent
0 stars 0 forks source link

Ability to stop making connections on seeding torrents #136

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Right now if I have nice ammount of torrents being seeded (e.g. 10000 :)), and 
max_connections set to, e.g. 128 and ~15000 peers in each torrent -- libtorrent 
will try to make 128 connections every tick() second. It will connect to other 
seeders, get "upload_to_upload" error and will try next.

While in "safe" environments trying to connect peers in seeding mode is not 
very meaningful. All peers will 100% reach seeder machine themselves.

The only reason for doing this -- few minutes after torrent been added: if I 
add torrent (seeding) in server1 and try immidiately to dl it from server2 
there is small chance server2 will reach tracker first. So, server2 will not 
get any peers from tracker, while server1 will. Right now server1 will connect 
server2.

So, desired functionality should allow me to put in/out torrents from 
"dont-make-connections" mode in runtime, but, obviously, allow to accept 
incoming connections (this why I cant just set max_connections=0 for this 
torrent).

You could make also ability to tune connection speed separately for seeding and 
downloading torrents. As far as I can see design of session_impl.tick() this 
will require to change it design a little.

E.g. set connection_speed = 100, and seed_connection_speed = 5. This could try 
to make 100 connections each tick for downloading torrents only. And if it will 
make <5 connections -- try to make connections from seeding torrents untill 
connection counter in this tick will reach 5.

Making meaningless connections is quite CPU hungry (comparing to not making 
connections :))

Another thing I need to notice -- in safe environments we could trust 
seeds/peers list come from tracker. And never try to connect seeds if we are 
seeder itself. This also might be tunable.

Original issue reported on code.google.com by mocksoul on 14 Dec 2010 at 7:34

GoogleCodeExporter commented 9 years ago 
I think I like the per-torrent connect speed. It sounds like that would be the 
most generic setting.

I don't understand your last paragraph though. I would imagine that in a safe 
environment you would control the tracker, and you would make it not send seeds 
to other seeds, but it sounds like that's not what you're talking about

Original comment by arvid.no...@gmail.com on 14 Dec 2010 at 6:44

GoogleCodeExporter commented 9 years ago 
See: how seeding peer can get in our torrent peerlist if we are seeding too:
1) from tracker
2) from another peer via PEX
3) maybe this peer was not seed few minutes ago, but completed download after 
it came to our peer list.

While 3rd case is completely normal -- we obviously will need to connect that 
peer, others not. As far as I can see -- PEX message is constructed for torrent 
once per minute and will be sended to all peers, not matter seeders they or 
not. Thus, obviously some seeds will get another seeds from PEX. And also as 
far as I can see right now -- on_extended() just adds them, not matter if this 
machine already seeds this torrent.

Also peers could come from some other exotic sources, like connect_peer() 
called by me :).

In safe setups, I guess, seeder list on whole network is much much bigger than 
leecher list. So leechers could be much more active than seeders, while latter 
could try to connect somebody too, but not that often.

In my setup, if somebody gives me a peer and says what it is a seeder (either 
tracker, or PEX) -- this is 100% means that it IS seeding machine. And if I'm 
seeder too -- best thing I can do is just ignore that peer this time.

On the other hand -- even in non-safe setups connecting to seeder from seeder 
is not good idea in my vision. If somebody "lies" and actually is not a seeder 
-- it his own problem and he will not get anything from swarm :-P

btw, decreasing failcnt factor after we get peer from tracker should also be 
tunable (right now you decrease failcnt by 1 after peer got from tracker again. 
In my setup reseting to 0 is preferable).

Original comment by mocksoul on 14 Dec 2010 at 7:29

GoogleCodeExporter commented 9 years ago 
I realize there are many ways for seeds to get into your peer-list when you're 
a seed, I would assume that it would be fairly easy to make the tracker not to 
be one of them.

The peer lists from trackers do not contain additional information on whether 
or not a peer is a seed. PEX messages do however, and I believe this 
information is used to not attempt connections to seeds when you're a seed. At 
least that's the intention.

Would you mind explaining how resetting the fail count to 0 would help? I would 
imagine your peers would barely ever fail, no?

Regarding the topic of this ticket, right now, when we close a connection 
because we're both seeds, I believe the peer is marked as being a seed and 
we'll never connect to it again. The fact that you've noticed seeds connecting 
to too many peers suggests that you have a lot of peers to go through then, 
because presumably you only connect to them once each, right?

Original comment by arvid.no...@gmail.com on 14 Dec 2010 at 10:10

GoogleCodeExporter commented 9 years ago 
failcounter decrease actually not related to this ticket, that was just thought 
I'll explain later.

About connections: I see my servers *always* trying to connect somebody. That 
seems that there are always 128 conn/s second. Why?

1) I have ~15000 servers
2) I have totally ~100000 torrents, but average spread is ~2000 torrents per 
server
3) 99% of peers is seeders as of now.

So, 1000 torrents with 1000 peers each makes totally 1 million peers. So, if we 
will try to connect them 100 times per second -- that will require 10000 
seconds to complete, or 3 hours. But also I have tracker interval set to ~3 
hours -- and probably after receiving new peers from tracker libtorrent will 
try to connect them again. And they will sent their new peers via PEX.. and so 
on so on.

I'm afraid what all these bunch of connection is not needed actually. Tracker 
(at least udp variant) sends separate complete/incomplete lists, isnt it? But 
because of PEX making tracker to send only peers to seeds will not make much 
sense -- to be honest your PEX implementation works like a charm -- ~5 minutes 
and almost all servers downloading or sharing one torrent know each other =).

I'm already tried to disable any new connections in seeding mode completely. 
That works, but -- as I already stated -- problem "leecher could connect 
tracker faster than seeder" appeared.

I'm also should notice what in such safe environments libtorrent (and whole 
torrrent protocol) used to speedup spreading huge amounts of data across 
servers. In this point of view leechers are much much more important than 
seeders. And this why it is very good idea (in my view) to make different 
limits for them:

1. Separate conn/s for seeding and downloading torrents
2. Separate max_failcount for leechers and seeders (e.g. try to c onnect 
leechers more often and more times)

Also - dont think what peer might fail if something happened this him. This is 
usually 1% of peer connect failures (at least in my case). More often these 
errors come from usual network errors (e.g. packets died in routers, switches, 
etc.). And these problems usually either dissapear themselves, either will be 
solved in ~10 minutes or more. So, if we have failcount 3 for peers -- we will 
make 3 attempts to connect in less than 1 minute. After that -- we will try 
connect this peer only if it comes again from tracker or PEX. But if we have a 
LOT peers in swarm -- than we have quite little chance this happen soon. So, 
that could be good idea to connect him again. But how? This thought, again, not 
100% related to this ticket, I just want to talk with you a lot :)

Original comment by mocksoul on 14 Dec 2010 at 11:02

GoogleCodeExporter commented 9 years ago 
This is a very simple patch to make seeding (and finished) torrents not make 
outgoing connections. Could you let me know if it works?

to enable it set session_settings::seeding_outgoing_connections = false;

The patch is against trunk.

Original comment by arvid.no...@gmail.com on 16 Dec 2010 at 3:20

Attachments:

GoogleCodeExporter commented 9 years ago

Original comment by arvid.no...@gmail.com on 16 Dec 2010 at 8:49

GoogleCodeExporter commented 9 years ago 
Obviously it *will* work, coz I did already tried to put into want_more_peers() 
simple "&& !is_seed()" call. But the only problem I had with that -- case then 
you want to upload something from serverA to serverB and serverB will reach 
tracker *before* serverA. Thus, serverA will have 1 leecher from tracker, while 
serverB will have 0 peers from tracker.

I'll deal with this by rerequesting peers for leecher machine in case it got 0 
from server sometimes. I'll need this because also if we got seeder from server 
-- that does not meen that seeder will be alive untill we download everything 
from him :).

So, this option will be useful for me, but I'll test it soon (in 1 week or so), 
so, please wait closing this ticket :)

Original comment by mocksoul on 16 Dec 2010 at 10:51

GoogleCodeExporter commented 9 years ago 
I saw you usually working in branches and after it is finished you merge it to 
trunk. So, trunk should be somewhat stable. Is that true? I'm currenly sitting 
with 0.15.4 + patchset.

Original comment by mocksoul on 16 Dec 2010 at 10:52

GoogleCodeExporter commented 9 years ago 
trunk is somewhat stable. I do larger features in separate branches, often I 
just put them in trunk, and discover bugs in them a week later or so, and fixes 
them.

I'm hoping to be able to release 0.15.5 soon, and then I'll probably branch off 
0.16.0 from trunk for stabilization.

Original comment by arvid.no...@gmail.com on 16 Dec 2010 at 11:31

GoogleCodeExporter commented 9 years ago 
Ok, probably I will help.

Right now I'm still sticking with 0.15.4 branch, still stabilizing all my own 
stuff to work properly with libtorrent.

After that (I guess, right after World will hit 2011 ;)) -- I'll try to check 
your RC0.15 branch and trunk. I have my own test suite and since I'm using 
libtorrent very heavily -- errors will pop up quite fast (if any =)).

Original comment by mocksoul on 16 Dec 2010 at 11:41

GoogleCodeExporter commented 9 years ago 
I checked in that patch to trunk

Original comment by arvid.no...@gmail.com on 17 Dec 2010 at 3:21

GoogleCodeExporter commented 9 years ago 
Actually, this does not seem to work somehow.

&& !is_seed() in torrent.cpp (my variant) works, but && 
m_ses.settings().seeding_outgoing_connections || m_state != 
torrent_status::seeding || m_state != torrent_status::finished -- does not 
(yep, I did set seeding_outgoing_connections to false by default, of course :))

Original comment by mocksoul on 17 Dec 2010 at 4:58

GoogleCodeExporter commented 9 years ago 
Ok, obviously. this will be proper working variant:

&& (m_ses.settings().seeding_outgoing_connections
    || (m_state != torrent_status::seeding
        && m_state != torrent_status::finished))

Original comment by mocksoul on 18 Dec 2010 at 1:05

GoogleCodeExporter commented 9 years ago 
that's a good point. fixed. Should I leave this ticket open for the other issue 
you mentioned? that the tracker announce interval may still screw things up?

Original comment by arvid.no...@gmail.com on 18 Dec 2010 at 11:20

GoogleCodeExporter commented 9 years ago

Original comment by arvid.no...@gmail.com on 17 Apr 2011 at 8:41