Deluge not agressive enough during initial (first) tracker update routine

[GoogleCodeExporter]

Greetings.

What's described below, has been opened as a feature request over at Deluge's 
ticket system:
http://dev.deluge-torrent.org/ticket/2059

But instead, I've been asked to post it here so here goes:

I use autodl-irssi to match the latest releases in my tracker.

Here's what happens:
1. My autodl is set to pass on .torrent files after 35 seconds to Deluge's 
watch-dir.

2. Deluge picks up the .torrent.

3. Deluge tries to update for the tracker once.

4. If the uploader of said torrent didn't perform "update tracker" on his 
client in a timely manner - Deluge will time-out and set itself to retry once 
more in no less than 30 minutes!

5. Those first 30 minutes, in SeedBox?-heavy private trackers, could mean you 
either:

5. A. Scored big: if Deluge managed to sync up with the tracker and you caught 
the swarm early.

5. B. Failed miserably: if Deluge timed-out for 30 minutes on the first 
announcement, and you managed to destroy your ratio on a particular torrent.

This is something that doesn't happen in rTorrent/ruTorrent - where it seems to 
keep on auto-retrying in very small intervals until it's able to sync up with 
the tracker.

And so, I kindly ask you implement a more aggressive initial tracker-updating 
mechanism, just until Deluge is able to pick up and sync with the tracker.

Something in the range of every 5 seconds would be very good. Again, just until 
it syncs up with the tracker.

BTW: increasing the upload-delay-secs = to anything higher than 35 seconds is 
quite futile, since there could always be those odd torrents where even after 
90 seconds they'd still won't sync, and after 150-180 seconds they would..

Specs:
Deluge Client: v1.3.4
Deluge Server: v1.3.4
libtorrent: v0.15.10.0
OS: Linux version 2.6.39.2-grsec-xxx-r1.5.1 (root@…) (gcc version 4.4.3 
(Gentoo 4.4.3-r2 p1.2) ) #1 SMP Wed Jun 29 00:47:56 EDT 2011

Regards,

Original issue reported on code.google.com by pcfregis...@gmail.com on 21 Mar 2012 at 1:36

[GoogleCodeExporter]

Hi. This is something people have discussed with me a few times in the past. My 
main concern with doing this the naive way, is that it opens up for serious 
DDoS capabilities in libtorrent.

Imagine someone posting a torrent claiming to be the latest block-buster movie, 
but instead it contains trackers to a few web sites. Everyone downloading this 
torrent will instantly join a swarm of DDoSers, hammering those sites every 5 
seconds, indefinitely (or until everyone realizes it's a scam and removes the 
torrent).

Now, that said, is there any distinguishing features of the tracker failure in 
your case? Is there any way to recognize the tracker as being in this state 
where it makes sense to retry? To avoid allowing general DDoSing.

Also, if we can solve this issue, do you still need those 35 seconds? It seems 
like those could potentially be 35 valuable seconds.

Original comment by arvid.no...@gmail.com on 21 Mar 2012 at 4:42

[GoogleCodeExporter]

Hi Arvid.

Thanks for getting back to me.
1. Although you could recognize the tracker by its name, but I believe that for 
the sake of Deluge/libtorrent keeping and gaining a prominent lead over other 
torrent software solutions - it should just be set as a global feature.

2. You're welcome to drop me an Email if you wish receive the private tracker's 
name.

3. If you could solve this issue - then the need for those 35 seconds would 
indeed become null and void, Again: Provided that libtorrent keeps on retrying 
every 5 seconds until it's able to establish the _initial_ "sync-up" with the 
tracker.

Regards,

Original comment by pcfregis...@gmail.com on 21 Mar 2012 at 8:09

[GoogleCodeExporter]

are there any other traits you can think of that would be more generic than a 
specific hostname? I would imagine that this is a somewhat wider problem than 
just this one tracker.

What's the tracker response for instance?

Original comment by arvid.no...@gmail.com on 22 Mar 2012 at 1:50

[GoogleCodeExporter]

Hello.

Sorry, I can't think of a specific Tracker trait, but indeed the observed 
behavior described in my first post is the same every time that:
1. An Uploader gets his torrent listed on whichever tracker.

2. The said .torrent has been listed and been made available to download to the 
end-user's client of choice via the Tracker's web-site / IRC channel  _before_ 
the Uploader actually performs the final 'force update tracker' (or what have 
you) on his client.

3. Once the .torrent has been loaded into Deluge, the response in the Tracker 
Stus line would be the same each and every time:
TrackerName: Error: unregistered torrent

4. Deluge would enter a time-out of 30 minutes before attempting to sync up 
with the tracker once more.

5. I think that the retry-attempts window should be:
5. A. Once every 5 seconds during the initial 300 seconds of failed sync.
5. B. Once every 15 seconds during the 300 seconds after that.
5. C. Once every 30 seconds during the 300 seconds after that.
5. D. Once every 60 seconds during the 900 seconds after that.
5. E. Once every 90 seconds during the next 1800 seconds.
5. F. Once every 1800 seconds after 5. A. - 5. E. have been exhausted.

I sincerely hope you would oblige and help remedy this behavior.

Regards,

Original comment by pcfregis...@gmail.com on 22 Mar 2012 at 7:20

[GoogleCodeExporter]

Hi,

I'd like to note, that in addition to my previous post from 6 hours ago, I've 
found that the retry-time-out value is something that is (and also can be?) 
controlled from the Tracker side and not from Deluge.

i.e: At least on one other private Tracker the sync retry-time-out value sits 
at around 32 minutes instead of 30.

In any case, that shouldn't affect the requested behavior described above.

Regards,

Original comment by pcfregis...@gmail.com on 22 Mar 2012 at 2:22

[GoogleCodeExporter]

You're right that the tracker controls the re-announce interval. In the case of 
a tracker knowing that a torrent is on its way, it could actually return an 
"interval" of just a few seconds. I would argue that the proper fix for this 
problem is, in order of properness:

1. the tracker site should not distribute the .torrent file before it's 
announced to the tracker
2. the tracker should respond with a short retry timeout in the case of a newly 
added torrent where the seed hasn't announced yet
3. the client should try to guess that this is the case and retry sooner, 
without a significant false positive rate, causing DDoSing of trackers.

Now, I'm open to suggestions on how to identify this case. Looking for "Error: 
unregistered torrent" seems somewhat error prone (what if the message is in 
another language, formatted slightly differently, etc.) and also covert a lot 
of other cases as well, where retrying soon is the opposite of what you want to 
do. i.e. the case where the torrent is in fact not registered with a tracker, 
and will never be.

Any other ideas?

Original comment by arvid.no...@gmail.com on 23 Mar 2012 at 8:34

[GoogleCodeExporter]

I thank you for the explanation, and understand the priority structure you've 
described.

Unfortunately, I don't have any additional ideas as how to isolate these 
sync-time-out occurrences in terms of error-report parsing on Deluge's side.

I can only hope you'll agree to what's been described in post #4 
(http://code.google.com/p/libtorrent/issues/detail?id=301#c4),
as it seems to be something that's already implemented in another product, 
which to my opinion - is pretty far off from Deluge's capabilities in terms of 
aggressive peering (which is everything in private-trackers).

Kindly let me know if this is something you're willing to give a go at, rather 
sooner than later.

Regards,

Original comment by pcfregis...@gmail.com on 23 Mar 2012 at 9:18

[GoogleCodeExporter]

could you provide a wireshark dump of the response, or just a verbatim copy of 
the response body (the whole bencoded message) you get when this happens?

I'm curious to know if the tracker includes the "interval" or "min_interval" 
field. If it does, it seems even scarier to disregard those and retry sooner.

Original comment by arvid.no...@gmail.com on 24 Mar 2012 at 1:33

[GoogleCodeExporter]

Could you try this patch and let me know if it makes a difference? (this is 
against trunk).

Original comment by arvid.no...@gmail.com on 24 Mar 2012 at 1:41

Attachments:

• tracker_timeout.diff

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi,

I sincerely appreciate the time you're taking into this!

Since I'm not leasing a dedicated server (it's a shared slot) and can't 
directly rely on my provider to cook specific updates for me - I've already 
setup Ubuntu 11.10 + Deluge v1.3.3 / libtorrent 0.15.7.0 as a local VirtualBox 
on my machine.

I'm currently able to connect to Deluge's daemon on the VM from my host.

I have some *very* basic knowledge with:
1. WireShark: I should be able to pass along the capture you've requeste. I 
just wish for its contents to remain private (i.e: for your eyes only), how can 
I do that?

2. *nix environments: Could you please tell me what will be the easiest route 
to have the changes you've put into tracker_timeouts.diff updated and reflected 
in my VM?

I'm terribly sorry to bother you with the above.

Regards,

Original comment by pcfregis...@gmail.com on 24 Mar 2012 at 10:14

[GoogleCodeExporter]

for the wireshark capture; you could either copy-paste the response only (from 
the follow-tcp-stream window), and make sure it doesn't include any info-hash 
or announce URL. The response typically doesn't include any of that, just the 
request. You could also email it directly to me at arvid@rasterbar.com, or both.

to apply the patch to a clean checkout of trunk: patch -p0 <tracker_timeout.diff

to build:

./autotool.sh
./configure --enable-python-binding
make

(there may be some other options you have to pass to ./configure, depending a 
bit on the distro. see ./configure --help for details)

Original comment by arvid.no...@gmail.com on 24 Mar 2012 at 2:49

[GoogleCodeExporter]

Original comment by arvid.no...@gmail.com on 30 Mar 2012 at 4:32

• Changed state: Fixed