Closed niayh closed 8 years ago
Upon a bit more investigation, this appears to be an issue localized to a single tracker. Is there any way to refine the definition of failure in this circumstance?
I would be interested in seeing an example response from this tracker, if you have one (ideally as a wireshark capture).
I would expect that if the tracker responds with a proper "failure reason" key, libtorrent would treat it as a failure (as opposed to a "warning message" key). However, whether libtorrent treats it as a fatal error or not probably would depend on some other property of the response. i.e. tracker failures aren't necessarily fatal, but could be transient. libtorrent does not implement http://bittorrent.org/beps/bep_0031.html
However, passing a very large announce interval might work.
Thanks for the response. I got you an example output log from deluge when the error occurs and I also grabbed 2 different sets of packets that I received in response from trying to add an unregistered torrent to the tracker.
https://www.dropbox.com/s/440db7d65ct64v6/unregistered.log?dl=0
Wireshark versions of responses: https://www.dropbox.com/s/5oudezfx4cl0omz/unregistered.pcap?dl=0
Do you mean increasing the autodl delay? I'm reluctant to do so because even with as large of a delay as 45 seconds, I still miss announces from time to time. Conversely, torrents that announce within 15 seconds, I would miss all or part of the main swarm.
Set up a few more autodl tests on other trackers and turns out another one of my trackers performs in the same manner to the one in the test, so that makes 2 trackers now.
by "announce interval" I meant the key the tracker is supposed to return in the tracker response. If the tracker would do that, libtorrent would not try again until that many seconds.
I notice the tracker response in your capture literally just contain a "failure reason" key/value. This is a bit odd and not really how I would read the specification. I would think that it's reasonable to try again 30 minutes later in this case. Is that what you see libtorrent do?
I'm curious, how do other clients behave? Do they simply consider the response invalid and never talk to the tracker again? (for that torrent). If so, I wonder if the reason not to talk to it again is that the response is too broken, or because the "failure reason" message exist in the response.
Ahh, I see now, what you mean by announce interval.
Libtorrent is indeed behaving how you think: resetting to 30 minutes
When I use rtorrent the behavior is what I am hoping to see in deluge: rtorrent lists the torrent as if it is in error mode and keeps trying to sync with the tracker in small intervals until it manages to do so. At this point, it backs of to regular intervals.
please re-open if there's anything actionable here
I'm also seeing this as well with libtorrent: 1.0.7.0 and deluge 1.3.12, where rtorrent will correctly retry in smaller intervals deluge sets the retry to 30 minutes which means you completely miss the swarm when it eventually announces.
Is it not possible to treat value received per the above captures as an error state that will trigger tracker_backoff ?
my understanding of what you're asking for is the opposite of back-off.. if a tracker fails, you want to try it again _morefrequently than you would normally contact it. The obvious risk is that a tracker getting overwhelmed with traffic will receive increasing amount of traffic as clients retry frequently.
Is there something unique about these tracker failures that indicate that the torrent will be available soon?
Do you have an idea of a reasonable behavior to improve the situation?
Yeah, not 100% sure on terminology, I'm just expanding on what the OP posted about.
I guess the unique aspect comes from using something like autodl-irssi which automatically downloads a torrent file that has been announced by a particular trackers bot.
What happens is the download of the torrent file happens so quick, then added to deluge, before the tracker seems to be aware of it so it ends up sitting in an unregistered state until the torrent client announces again. So in a sense the torrent file is definitely valid, just happens too fast.
With autodl-irssi you can set a delay to avoid this but as described above its a fine line between setting the delay for too long or too short that you end up missing the swam.
The way rtorrent handles the exact same tracker is it retries within seconds of the torrent being added in the "unregistered torrent" state, rather than waiting 30 minutes like Deluge
would it be reasonable to perhaps have an exception where, if the torrent was added recently enough, the retry timeout is shorter.
This sounds reasonable to me -- even a window as small as 5-10 minutes could be sufficient to avoid the described behavior
I think 5 to 10 minutes is still too long though, it needs to be in seconds as on competitive trackers 5 to 10 minutes is long enough for 90% of people to have completed the download which would result in a lower than acceptable ratio
Can it be a ramping sort of behaviour e.g. Try in 5 seconds then (if failed)10 seconds then 15 seconds and so on?
Obviously the goal is to not smash the tracker so I'm not sure how rtorrent does it
On Wednesday, February 10, 2016, niayh notifications@github.com wrote:
This sounds reasonable to me -- even a window as small as 5-10 minutes could be sufficient to avoid the described behavior
— Reply to this email directly or view it on GitHub https://github.com/arvidn/libtorrent/issues/315#issuecomment-182144018.
on the other hand, if the tracker admins incentivize hammering the tracker, and publish the torrent before their tracker whitelist it, perhaps they want people to hammer their tracker.
I'm not really sure what to say other than rtorrent handles it fine
Personally I'm opposed to making any accommodations for this. The trackers here are broken in at least two ways:
These issues should be taken up with the tracker admins, not hacked around in libtorrent.
Probably easier to just stick with adding a delay to the download script.
As you say, it's clearly a tracker issue.
On Wed, Feb 10, 2016 at 4:33 PM, Steven Siloti notifications@github.com wrote:
Personally I'm opposed to making any accommodations for this. The trackers here are broken in at least two ways:
- They publish torrents before they have been authorized.
- They enforce a minimum ratio. This is broken because pure ratio is a zero sum so you end up with the degenerate behavior of people jumping on any and all torrents the instant they go up in a desperate bid to find someone to upload to.
These issues should be taken up with the tracker admins, not hacked around in libtorrent.
— Reply to this email directly or view it on GitHub https://github.com/arvidn/libtorrent/issues/315#issuecomment-182182930.
your download script could hammer the tracker until receiving a legitimate response, and then add the torrent. :)
lol, what it does is when it receives the announce from the irc bot it waits X seconds before downloading the torrent file.. that way nothing gets hammered
On Wed, Feb 10, 2016 at 5:26 PM, Arvid Norberg notifications@github.com wrote:
your download script could hammer the tracker until receiving a legitimate response, and then add the torrent. :)
— Reply to this email directly or view it on GitHub https://github.com/arvidn/libtorrent/issues/315#issuecomment-182191689.
Actually, now that I've had more time to process this wouldn't you just be adding another condition based on the OP's scenario that would use the tracker_backoff function?
Or am I just being ignorant here?
trackers are re-tried with an exponential back-off if we fail to connect to them (and possibly if they return some fatal error at a layer below the bittorrent-tracker protocol, like an HTTP 503).
In this case, we are successful in connecting and talking to the tracker at the bittorrent-level protocol. This means we can be certain the tracker is there and expects to be used as a bittorrent tracker.
At that point, the tracker has a fair amount of control over the client's behavior (for good reasons). If the tracker wants the client to never connect back again, or try again in a few seconds, it can ask the client to do so. In this case, iirc, it returns a somewhat strange and ambiguous response.
(also, currently there's actually a sanity check in libtorrent clamping re-announce intervals to 5 minutes, maybe the tracker should be trusted more than that).
My point is that, yes, it's possible to add a case where this is considered a failure at a lower level, and the tracker is retried. But this is just one tracker. The next tracker will presumably have a slightly different response in this situation. I imagine it would make a lot more sense to tighten up the specification and stay more true to it, than to sprinkle special cases around. The risk is that, at some point you run into a case where the special case causes the opposite effect than you want, and the user/admin is surprised of the behavior because it's undocumented.
Fair call.
Thanks for the explanation, makes more sense now!
On Thu, Feb 11, 2016 at 4:12 AM, Arvid Norberg notifications@github.com wrote:
trackers are re-tried with an exponential back-off if we fail to connect to them (and possibly if they return some fatal error at a layer below the bittorrent-tracker protocol, like an HTTP 503).
In this case, we are successful in connecting and talking to the tracker at the bittorrent-level protocol. This means we can be certain the tracker is there and expects to be used as a bittorrent tracker.
At that point, the tracker has a fair amount of control over the client's behavior (for good reasons). If the tracker wants the client to never connect back again, or try again in a few seconds, it can ask the client to do so. In this case, iirc, it returns a somewhat strange and ambiguous response.
(also, currently there's actually a sanity check in libtorrent clamping re-announce intervals to 5 minutes, maybe the tracker should be trusted more than that).
My point is that, yes, it's possible to add a case where this is considered a failure at a lower level, and the tracker is retried. But this is just one tracker. The next tracker will presumably have a slightly different response in this situation. I imagine it would make a lot more sense to tighten up the specification and stay more true to it, than to sprinkle special cases around. The risk is that, at some point you run into a case where the special case causes the opposite effect that you want, and the user/admin is surprised of the behavior because it's undocumented.
— Reply to this email directly or view it on GitHub https://github.com/arvidn/libtorrent/issues/315#issuecomment-182417782.
Hi. I had the same problem. I found the following solution and it works.
http://wiki.bytesized-hosting.com/wiki/index.php/Autodl-irssi
It is the upload-delay-secs parameter. I tried 27 as stated but it didn't work. I the tried as 60 and it worked. I have not fine tuned to find the lowest number of seconds. This parameter can be set by tracker in the autodl.cfg file, and there is a global Delay parameter on the Filter General tab. I set 60 on both. Thanks for identifying this problem, I have been searching for days for a solution. Jack.
Oops I should have said I am using the rutorrent gui for the autodl-irssi configuration and it now works for both rtorrent and deluge so I set the global Delay parameter on rutorrent and I changed the ipt section on the deluge autodl.cfg for the tracker modification.
Let me try one more time. I guess the autodl.cfg is for the audodl-irssi. On the rutorrent autodl-irssi Filter General tab there is a Delay at the global level and on the Trackers tracker tab there is a Delay at that tracker level.
Yeah, the delay functionality is useful.
The only problem I have setting a delay is there's a chance you might miss the initial swarm. Unfortunately the tracker in question will announce within a few seconds or more than a minute so there's quite a variance there.
In the end I obtained and modified a script which makes use of the Execute plugin for Deluge and update-tracker.py command from here https://github.com/s0undt3ch/Deluge/blob/master/deluge/ui/console/commands/update-tracker.py
It loops through looking for the word unregistered and will initiate an update-tracker command via deluge-console till it gets an Announce OK.
It's not ideal but it gets around the variable announce
On Wed, Feb 24, 2016 at 8:17 AM, jmo4114979 notifications@github.com wrote:
Let me try one more time. I guess the autodl.cfg is for the audodl-irssi. On the rutorrent autodl-irssi Filter General tab there is a Delay at the global level and on the Trackers tracker tab there is a Delay at that tracker level.
— Reply to this email directly or view it on GitHub https://github.com/arvidn/libtorrent/issues/315#issuecomment-187849405.
another interesting approach would be to remember the peers you typically see early on in swarms on a specific tracker, and optimistically just add those to the swarm. you may be lucky and hit the original seed even before the tracker gets around to whitelisting
@slaughtermouse could you... post the script you 'obtained'?
Cough... thanks.
#!/bin/bash
torrentid=$1
torrentname=$2
torrentpath=$3
x=1
while [ $x -le 45 ]
do
sleep 2
echo "Running $x times" >> script.log
echo "TorrentID: $torrentid" >> script.log
line=$(/usr/local/bin/deluge-console "connect 127.0.0.1:<port> <user> <pass>; info" $1 | grep "Tracker status")
echo $line >> script.log
case "$line" in
*unregistered*|*Sent*)
eval /usr/local/bin/deluge-console "connect 127.0.0.1:<port> <user> <pass>\; update-tracker '$torrentid'";;
*)
echo "Found working torrent: $torrentname $torrentpath $torrentid" >> script.log
exit 1;;
esac
x=$(( $x + 1 ))
done
Save it to a file of your choice, use the Execute plugin in deluge. You'll need to modify the port, user pass to suit
It's crude, some of the things in there are superfluous, but does the job
slaughtermouse.. thanks for the tips. I've been having the same issue with a few trackers lately. I am using autodl with deluge. I am missing the swarm and end up the last leecher thus messing up my ratio for any specific torrent for which i have set manual delay.
With the above work around, please let me know if i did the right thing, or i need to fix the steps.
Is this the correct steps?
Yep that's correct
You might need to restart deluge once you've enabled and added the execute script once started.
When you add a torrent you should get a script.log generated in the same location as the script
On Tue, Jun 7, 2016 at 1:55 AM, monukoshy notifications@github.com wrote:
slaughtermouse.. thanks for the tips. I've been having the same issue with a few trackers lately. I am using autodl with deluge. I am missing the swarm and end up the last leecher thus messing up my ratio for any specific torrent for which i have set manual delay.
With the above work around, please let me know if i did the right thing, or i need to fix the steps.
- created a scritpt workaround.sh
- added the code to this script, and edited port and username/password entries for two different locations mentioned in the code.
- enabled the execute plugin, and in the drop down i choose, "Torrent Added" and in the command line box i type "sh /home/user/.config/deluge/workaround.sh"
- Hit apply and now wait.
Is this the correct steps?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/arvidn/libtorrent/issues/315#issuecomment-223965925, or mute the thread https://github.com/notifications/unsubscribe/AOQZ3S6wunnfBSsW_69MiP5kcvFl-iH5ks5qJCa-gaJpZM4GvytU .
@slaughtermouse , I am pretty new to unix and my seedbox, I'm trying to implement your script using the path and script name monukoshy provided.
This is the correct command and path for execute? sh /home/user/.config/deluge/workaround.sh I get an execute script not found or not executable error when adding a new torrent
I am also not sure what port I should be putting into the script.
Thanks
Yes, you have to restart deluge after you've enabled execute and added the script.. as I've chmod +x'd my script I just call it as /home/user/script
The port refers to the port your deluge instance is listening on, default is 20000 I think?
On Fri, Jun 10, 2016 at 2:21 PM, plasticspoonn notifications@github.com wrote:
@slaughtermouse https://github.com/slaughtermouse , I am pretty new to unix and my seedbox, I'm trying to implement your script using the path and script name monukoshy provided.
This is the correct command and path for execute? sh /home/user/.config/deluge/workaround.sh
I am also not sure what port I should be putting into the script.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/arvidn/libtorrent/issues/315#issuecomment-225078498, or mute the thread https://github.com/notifications/unsubscribe/AOQZ3efQCphpwDx5_A_h-cxwaj5Zi9L_ks5qKMopgaJpZM4GvytU .
@slaughtermouse, Still getting same error, if I manually go into that folder and do a sh workaround.sh I can get it to spit out the script.log. I think deluge cannot find the path. This is the path of the file for me
/media/sdj1/
I have tried that and the path /home/user/.config/deluge/workaround.sh provided.
As for the port, is it the server interface port? not the daemon port I assume.
Daemon port yes
Not sure, put the script in /media/sdj1 instead and/or try without sh
On Friday, 10 June 2016, plasticspoonn notifications@github.com wrote:
@slaughtermouse https://github.com/slaughtermouse, Still getting same error, if I manually go into that folder and do a sh workaround.sh I can get it to spit out the script.log. I think deluge cannot find the path. This is the path of the file for me
/media/sdj1//.config/deluge/workaround.sh
I have tried that and the path /home/user/.config/deluge/workaround.sh provided.
As for the port, is it the server interface port? not the daemon port I assume.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/arvidn/libtorrent/issues/315#issuecomment-225084290, or mute the thread https://github.com/notifications/unsubscribe/AOQZ3bCbJVHeitWq9FiQYho36HmTOC2rks5qKNfJgaJpZM4GvytU .
I've been seeing this issue quite often at multiple trackers and missing the initial swarm in todays competitive enviroment is a hard hit on ones ratio. @slaughtermouse, Your script looks like the only solution currently but after trying to convert it to batch for Windows and failing I thought I'd hear with you if that is something you've already done?
@MutatedHero, it's not something I've tried on Windows sorry.
But when I have some time I'll see if I can whip something up
Heya @slaughtermouse I set up your script by inserting my own info, but this is the error I get when I try to run it through putty on my seedbox.
/.config/deluge/delaything.sh: line 5: $'\r': command not found
/.config/deluge/delaything.sh: line 14: syntax error near unexpected token `$'in\r''
'/.config/deluge/delaything.sh: line 14: `case "$line" in
Any idea what that's all about? I have very little programming and linux skills so I can follow along with what the script is basically trying to do but any syntax details go right over my head.
No errors or any other indicators in deluge and no log is created. I'm on latest versions of everything. Thanks
@fluxcapacitus try editing in nano and remove the \r's from the end of each line in the script
From: fluxcapacitus notifications@github.com Sent: Tuesday, 17 January 2017 3:24:46 p.m. To: arvidn/libtorrent Cc: James Leonard; Mention Subject: Re: [arvidn/libtorrent] Error state on unregistered torrents not set correctly (tracker_backoff and autodl) (#315)
Heya @slaughtermousehttps://github.com/slaughtermouse I set up your script by inserting my own info, but this is the error I get when I try to run it through putty on my seedbox.
/.config/deluge/delaything.sh: line 5: $'\r': command not found
/.config/deluge/delaything.sh: line 14: syntax error near unexpected token $'in\r'' '/.config/deluge/delaything.sh: line 14:
case "$line" in
Any idea what that's all about? I have very little programming and linux skills so I can follow along with what the script is basically trying to do but any syntax details go right over my head.
No errors or any other indicators in deluge and no log is created. I'm on latest versions of everything. Thanks
- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/arvidn/libtorrent/issues/315#issuecomment-273005667, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AOQZ3Um6bq_wMBzjjvJ8b8imYNDwuEm3ks5rTCZugaJpZM4GvytU.
@slaughtermouse I'm not seeing any of those in the script, the only \ I even see is at the end of the second password placeholder but deleting that didn't stop the error.
torrentid=$1
torrentname=$2
torrentpath=$3
x=1
while [ $x -le 45 ]
do
sleep 2
echo "Running $x times" >> script.log
echo "TorrentID: $torrentid" >> script.log
line=$(/usr/local/bin/deluge-console "connect 127.0.0.1:<port> <user> <pass>; info" $1 | grep "Tracker status")
echo $line >> script.log
case "$line" in
*unregistered*|*Sent*)
eval /usr/local/bin/deluge-console "connect 127.0.0.1:<port> <user> <pass>\; update-tracker '$torrentid'";;
*)
echo "Found working torrent: $torrentname $torrentpath $torrentid" >> script.log
exit 1;;
esac
x=$(( $x + 1 ))
done
It's an exact paste of your code snippet with the placeholders replaced with my login and port. Was I supposed to do something with the code from the page you linked? https://github.com/s0undt3ch/Deluge/blob/master/deluge/ui/console/commands/update-tracker.py
Try typing: which deluge-console
The location of your deluge-console might be different to mine being in /usr/local/bin/
Also, probably not required but don't forget to include #!/bin/bash
at the top of the script
You need to copy update-tracker.py to the following, or relevant to your install, location: /usr/local/lib/python2.7/dist-packages/deluge-1.3.13_dev-py2.7.egg/deluge/ui/console/commands
The update-tracker.py file adds the update-tracker command to deluge-console, without it the script won't work
Failing that, here's the script (slightly updated since I originally posted it) https://drive.google.com/file/d/0B-Mod1AY73DdNTh6N1hiV25MY0E/view?usp=sharing
The problem was no tracker-update.py and I can't add it because I don't have the permissions for that location with this seedbox provider. I wasn't really pleased with the service anyway so this is enough to get me to switch. Thanks for the help, I'll come back in a month when this term expires.
On January 17, 2017 1:44:18 AM MST, slaughtermouse notifications@github.com wrote:
Try typing: which deluge-console
The location of your deluge-console might be different to mine being in /usr/local/bin/
Also, probably not required but don't forget to include
#!/bin/bash
at the top of the scriptYou need to copy update-tracker.py to the following, or relevant to your install, location: /usr/local/lib/python2.7/dist-packages/deluge-1.3.13_dev-py2.7.egg/deluge/ui/console/commands
Failing that, here's the script slightly updated https://drive.google.com/file/d/0B-Mod1AY73DdNTh6N1hiV25MY0E/view?usp=sharing
-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/arvidn/libtorrent/issues/315#issuecomment-273054181
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Script is working good for me. I initially got errors in both files until I made sure to remove tabs in .py file and and non-unix line endings in script. Thanks slaughtermouse.
@slaughtermouse Hi there, i have done everything right but the question is the Script just doesn't work whenever a new torrents added in watch folder with "unregistered torrents" . Any idea how to fix that? I have copy the update-tracker.py also enabled the Excute in Deluge, just doesn't work.
Normally you need to restart deluge as the execute plugin doesn't always work off the bat
Also I assume you've added the script to the execute plugin, selecting the "torrent added" event and the command as /home/
For good measure chmod +x trachk from the command line
@slaughtermouse I Do did all those. But it still does not work ? May you give a little test to see if it still works?
I have it running all the time
The script runs fine when you manually run it with no arguments?
Try running deluge-web without forking to daemon you might be able to see an error as to why the script isn't running when the torrent is added
On Fri, 5 May 2017 at 9:14 PM, HYQ307 notifications@github.com wrote:
@slaughtermouse https://github.com/slaughtermouse I Do did all those. But it still does not work ? May you give a little test to see if it still works?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/arvidn/libtorrent/issues/315#issuecomment-299418579, or mute the thread https://github.com/notifications/unsubscribe-auth/AOQZ3X5hGYG1NUZstBQcqyO944tkqHYYks5r2uhmgaJpZM4GvytU .
I manually run the script and get this: "Running 1 times TorrentID: Tracker status: xxx: Announce OK Tracker status: xxxx: Announce OK Tracker status: chdbits$ Found working torrent:"
So i guess the script works fine. But the question is whenever a new torrent added, it just doesn't work to update the tracker. Here is what i set in Deluge Excute: https://picload.org/image/rcpclcgl/1.png
I also reboot to make it apply. But doesn't work when an unregistered torrents added.
And how to do this "Try running deluge-web without forking to daemon you might be able to see"
You mean deluge-console like this?
https://picload.org/image/rcpclciw/1.png
@slaughtermouse
Not sure, all I can suggest is you don't need to add bash in front of the script, it should just be /location/to/script.sh
http://dev.deluge-torrent.org/wiki/Plugins/Execute
On 6 May 2017 at 18:54, HYQ307 notifications@github.com wrote:
I manually run the scripte and get this: "Running 1 times TorrentID: Tracker status: awesome-hd.me: Announce OK Tracker status: awesome-hd.me: Announce OK Tracker status: chdbits$ Found working torrent:"
So i guess the script works fine. But the question is whenever a new torrents added, it just doesn't work to update the tracker. Here is what i set in Deluge Excute: [img]https://picload.org/image/rcpclcgl/1.png[/img]
And i also reboot to make it apply. But doesn't work when an unregistered torrents added.
@slaughtermouse https://github.com/slaughtermouse
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/arvidn/libtorrent/issues/315#issuecomment-299620441, or mute the thread https://github.com/notifications/unsubscribe-auth/AOQZ3dj90MhBZU-IQ6aY5qOtUPtth3a5ks5r3Bk_gaJpZM4GvytU .
Yeah, this time works. Thansk for helping.
Hey there,
I've been trying to get Deluge 1.3.12 + Libtorrent 1.0.7 to work in conjunction with an autodl filter and a watch folder from a private tracker. I've been running around in circles trying to figure out why if when a torrent is added before the tracker has registered it, the client will wait a full 30 minutes until the next announce to refresh.
I found the tracker_backoff setting and was hopeful that this would fix my problem; however, no amount of changing this value would do anything to solve my problem with unregistered torrents. The torrent gets added and if the torrent is not registered, the announce interval is set to 30 minutes.
I started wondering, perhaps a failure isn't being reported properly somehow. Sure enough, I altered the tracker URL to something bogus and suddenly the retry intervals were set according to the tracker_backoff setting ( 7, 15, 27, 45, 95, 127, 165 seconds, etc). This leads me to believe that somehow the error status on unregistered torrents isn't being recognized properly, even though the Announce Status reads: Error: (Unregistered Torrent) and as a result of being able to connect to the tracker, libtorrent decides to announce as if all is normal and set the next refresh to 30 minutes.
This may be a tracker specific issue; however, it's worth noting that the error state is read correctly when using AutoDL with rtorrent.