Possibility of short announce intervals for racing torrents

[fusk-l]

So i've been away from qBit for a while. Reasons being read/write performance for 1gbit connections. We've talked a lot about this on the forum, and the best suggestion was to try something else. I've been running deluge for a while, it's fine but is not being updated for windows.

But deluge can do some things that i would not be able to live without.

1. A way/option to manually set libtorrent config, and having that config overwrite settings in qBit, allowing to set settings not available through the UI. Like write_cache_line_size, close_file_interval, contiguous_recv_buffer, cache_size_volatile etc.

2. In deluge i have a script running performing:

   
   #
   # core.py
   #
   # Basic plugin template created by:
   # Copyright (C) 2008 Martijn Voncken <mvoncken@gmail.com>
   # Copyright (C) 2007-2009 Andrew Resch <andrewresch@gmail.com>
   # Copyright (C) 2009 Damien Churchill <damoxc@gmail.com>
   #
   # Deluge is free software.
   #
   # You may redistribute it and/or modify it under the terms of the
   # GNU General Public License, as published by the Free Software
   # Foundation; either version 3 of the License, or (at your option)
   # any later version.
   #
   # deluge is distributed in the hope that it will be useful,
   # but WITHOUT ANY WARRANTY; without even the implied warranty of
   # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
   # See the GNU General Public License for more details.
   #
   # You should have received a copy of the GNU General Public License
   # along with deluge.    If not, write to:
   #    The Free Software Foundation, Inc.,
   #    51 Franklin Street, Fifth Floor
   #    Boston, MA  02110-1301, USA.
   #
   #    In addition, as a special exception, the copyright holders give
   #    permission to link the code of portions of this program with the OpenSSL
   #    library.
   #    You must obey the GNU General Public License in all respects for all of
   #    the code used other than OpenSSL. If you modify file(s) with this
   #    exception, you may extend this exception to your version of the file(s),
   #    but you are not obligated to do so. If you do not wish to do so, delete
   #    this exception statement from your version. If you delete this exception
   #    statement from all source files in the program, then also delete it here.
   #

import time import logging from twisted.internet.task import LoopingCall

from deluge.log import LOG as log from deluge.plugins.pluginbase import CorePluginBase import deluge.component as component import deluge.configmanager from deluge.core.rpcserver import export from deluge.ui.client import client

DEFAULT_PREFS = { "test":"NiNiNi" }

class Core(CorePluginBase):

timer = {}

def update_tracker(self, torrent_id):
    tid = component.get('TorrentManager').torrents[torrent_id]
    tid_status = tid.get_status(['tracker_status','time_added'])
    log.info("[AddTrackerCheck](%s)(%s) : %s", torrent_id, time.time() - tid_status['time_added'], tid_status['tracker_status'])

    if tid_status['tracker_status'].find("Announce OK") != -1:
        Core.timer[torrent_id].stop();
    elif time.time() - tid_status['time_added'] > 300:
        Core.timer[torrent_id].stop();
    else:
        time.sleep(5)
        log.info("[AddTrackerCheck](%s) : Updating Tracker", torrent_id)
        tid.force_reannounce();

def post_torrent_add(self, torrent_id, *from_state):
    if not component.get('TorrentManager').session_started:
        return
    log.info("[AddTrackerCheck](%s) : Adding New Torrent", torrent_id)
    Core.timer[torrent_id] = LoopingCall(self.update_tracker, torrent_id)
    Core.timer[torrent_id].start(2)

def enable(self):
    # Go through the commands list and register event handlers
    self.config = deluge.configmanager.ConfigManager("addtorrentcheck.conf", DEFAULT_PREFS)
    component.get("EventManager").register_event_handler("TorrentAddedEvent", self.post_torrent_add)

def disable(self):
    pass

def update(self):
    pass

@export
def set_config(self, config):
    """Sets the config dictionary"""
    for key in config.keys():
        self.config[key] = config[key]
    self.config.save()

@export
def get_config(self):
    """Returns the config dictionary"""
    return self.config.config

Any way to do in newer versions of qBit?

[FranciscoPombal]

@fusk-l

Reasons being read/write performance for 1gbit connections. We've talked a lot about this on the forum,

Can you link some of those threads here please? I would like to see what people over there are trying to do to get qBittorrent to work better under those circumstances.

I use qBittorrent is several machines, with and without gigabit connections with no issues. The only problem is that qBIttorrent's out-of-the-box config needs a bit of tweaking for it to achieve maximum performance, especially with very fast connections and/or many torrents.

From my experience, the most important thing to do is to set disk cache to -1 (auto) in the advanced settings. Hope this helps.

[fusk-l]

Just search the forum for "gbit" and you'll probably find what you're looking for. You have to remember 1gbit goes both ways, so simultaneous equals ~2gbit. Even a ssd raid0 would not be fast enough. Libtorrent config can be edited outside deluge, so even if the setting isn't available in UI, deluge will read and perform what the lib config dictates. This allows you to fine tune libtorrents cache, disk etc. config for high throughput. And it works, disk aren't overloading and my connection limit is higher than before. But you can read more about that on the forum.

[FranciscoPombal]

Just search the forum for "gbit" and you'll probably find what you're looking for.

Ok

You have to remember 1gbit goes both ways, so simultaneous equals ~2gbit. Even a ssd raid0 would not be fast enough.

I know that. In my previous reply I was already assuming operation under no (or slight) disk throughput bottlenecks.

Libtorrent config can be edited outside deluge, so even if the setting isn't available in UI, deluge will read and perform what the lib config dictates. This allows you to fine tune libtorrents cache, disk etc. config for high throughput. And it works, disk aren't overloading and my connection limit is higher than before. But you can read more about that on the forum.

Alright, I will try to get back to you after I read through it.

By the way, what about my suggestion of altering the disk cache setting? What has been your experience with it?

[xavier2k6]

@FranciscoPombal #11436 has a forum link that you can read up on/has some relevance...

[FranciscoPombal]

Ok so I did some reading and here are my conclusions:

1- Some benchmarking with very high disk and network throughput setups was done over in the forums; it was found that qBIttorrent/libtorrent are underperforming out of the box.

Currently, the hope is that the situation can be improved by tweaking some of libtorrent's advanced settings. Namely, the buffer watermark settings, settings like the above mentioned write_cache_line_size, contiguous_recv_buffer, cache_size_volatile and the new piece_extent_affinity.

However, so far the effects of these settings have not been comprehensively benchmarked in such a way that allows deducing generic configuration guidelines for different setups (fast/slow network, with fast/slow storage...). Plus, the full meaning and semantics of many of these settings are only known to very few people it seems.

Ideally if there should be more documentation and benchmarks of these settings, so that users can correctly configure their clients for maximum performance in a less hit-and-miss and error-prone way. Think of a wiki page or a settings calculator like this one.

2 - Regardless of point 1, I think qBittorrent needs to change some of its defaults. At least:

• DIsk cache: auto
• switching on the new piece_extent_affinity.

I think it is pretty clear these are safe and positive changes, even without the benchmarks to back it up.

3 - @fusk-l if I understand correctly, the deluge script you posted above is not strictly related to these matters; it is simply forcing the client to reannounce to the tracker in small intervals after being added, for the first 5 minutes. You are using this for "racing", right? I too think it would be nice if qBIttorrent had a setting that allowed enabling this behaviour.

For the uninitiated: When you are "racing" torrents, often the .torrent files are automatically obtained and added to the client a few seconds/minutes before the tracker has a chance to register them. Thus, the initial announce will fail. Unfortunately, most clients are configured to only automatically reannounce once every 30/45/60 minutes or so. This means that you will of course "lose" the race, unless you either manually intervene or have a script that manually reannounces within the first few moments after the torrent is added until the reannounce succeeds, as is the case.

[fusk-l]

3 - @fusk-l if I understand correctly, the deluge script you posted above is not strictly related to these matters; it is simply forcing the client to reannounce to the tracker in small intervals after being added, for the first 5 minutes. You are using this for "racing", right? I too think it would be nice if qBIttorrent had a setting that allowed enabling this behaviour.

Yes, that is correct. And while it might sound like a niche/special thing, as you mentioned there a trackers working like that. And as qbit has become a lot more popular over the past years, a lot more users are affected by it. So while it might only be useful for people on private trackers, that's still 10's of thousands of users out of the 100's of thousands that use qbit on private trackers. So i really do think this needs to be default behaviour in qbit.

Besides benchmarking, it was also found that qbit had excessive I/O's which was especially bad for ssd's, not just performance wise but also wear.

Libtorrent has a built in "high performance seed" profile which adjusts and enables settings previously disabled. While deluge has a plugin that can edit libtorrent config, i also think qbit needs something like that. I am not saying they should build a plugin module, but a way to access libtorrent config for advanced users. Maybe a button in advanced settings or something. That would basically make qbit perfect as deluge for windows development has stagnated and still running libtorrent 1.1.x and also doesn't support rss.

[FranciscoPombal]

Yes, that is correct. And while it might sound like a niche/special thing, as you mentioned there a trackers working like that. And as qbit has become a lot more popular over the past years, a lot more users are affected by it. So while it might only be useful for people on private trackers, that's still 10's of thousands of users out of the 100's of thousands that use qbit on private trackers. So i really do think this needs to be default behaviour in qbit.

Racing itself may be relatively niche, but the "initial announce problem" may be at the root of other long-standing issue reports, affecting other larger parts of the userbase.

For example, it just occurred to me that a lot of users use software that auto-adds torrents on release, such as sonarr or radarr. Even though these users are not racing, they will likely experience a 30/45/60 minute delay for the same reason. During this time, the torrent status is reported as "Stalled".

I wonder how many of the "stalled" issue reports, which are notoriously hard to reproduce, are actually caused by this. After all, many users use software such a radarr or sonarr. An example that is consistent with this analysis: https://github.com/qbittorrent/qBittorrent/issues/11375#issuecomment-549120105

@sledgehammer999 @Chocobo1 @glassez @thalieht What are your thoughts on this? If I am correct, then a large amount of the "stalled" issues can be fixed by forcing qBittorrent either to announce to the tracker more frequently in the first ~5 minutes after the torrent is added, or to announce more frequently in case of a bad tracker response. Or at the very least, report a new status like "waiting for good tracker response" instead of stalled if the previous tracker response is erroneous.

[Rouzax]

I wonder how many of the "stalled" issue reports, which are notoriously hard to reproduce, are actually caused by this. After all, many users use software such a radarr or sonarr. An example that is consistent with this analysis: #11375 (comment)

To comment on the stalled reference, in my case it stalls the entire client and will not download or seed anymore until I restart the client

[fusk-l]

To comment on the stalled reference, in my case it stalls the entire client and will not download or seed anymore until I restart the client

I think that's something unrelated to topic.

[glassez]

I think that's something unrelated to topic.

What's the topic? "[Wishlist] Want to return to qBit" is something meaningless to be here.

[fusk-l]

I think that's something unrelated to topic.

What's the topic? "[Wishlist] Want to return to qBit" is something meaningless to be here.

Whatever floats your boat.

[glassez]

the .torrent files are automatically obtained and added to the client a few seconds/minutes before the tracker has a chance to register them

Sorry, can't understand the reason. Why tracker hasn't a chance to register torrents? Can you explain it more detailed?

[FranciscoPombal]

@glassez

Sorry, can't understand the reason. Why tracker hasn't a chance to register torrents? Can you explain it more detailed?

Sometimes, the following happens: 1 - User uploads torrent to site. 1.5 (some seconds/minutes, asynchronously with respect to steps 2 and 3) - the associated tracker software registers the torrent that was added to the site. 2 - Site auto-distributes new torrent on IRC/RSS/whatever 3 - "Racer" clients auto-download the .torrent file as soon as it is available on the aforementioned channels and auto-start the download.

Sometimes step 3 may happen before step 1.5 is completed. In qBittorrent's case, this means that the first announce will fail (since the tracker doesn't yet know about the torrent) and the next announce will only happen 30-60 minutes later. Thus, it won't be able to "race", since after that much time all the racers will have most likely finished the download long ago. For qBittorrent to be able to race, it needs to (optionally) keep reannouncing every 5-10 seconds when announces fail within the first 5 minutes of adding the torrent, or something to that effect.

You may argue that in this case, this is a problem of the site/tracker operators. Perhaps they should only distribute the torrent after the tracker software has registered it. However, this problem actually affects a lot of users who happen to get a tracker error on the first announce. Those users will have their torrents stalled for at least the next 30-60 minutes. For instance, suppose I am using qBittorrent's embedded tracker to privately share a torrent with a friend and he adds it to the client before I start the embedded tracker. Then, unless my friend knows about the problem and does "force reannounce", he will think qbittorrent is broken.

Let me know if I need to clarify anything else.

[glassez]

If this feature (reconnect on failure) is widely used or really needed generally, then maybe it would be better if it was implemented inside the libtorrent? E.g. through some configurable option (such as "number of reconnections on failure"). @arvidn, what do you say?

[arvidn]

I'm not done reading through all of this thread yet, but does "reconnect" refer to re-announcing to a tracker? If so, does anyone have a good understanding why trackers fail so frequently that this is an issue? are they simply overrun by traffic and don't have sufficient capacity?

[FranciscoPombal]

@arvidn

Yes, I believe @glassez used the term "reconnect" referring to re-announcing to a tracker.

I am not sure as to why this is an issue. But it does happen in a lot of trackers; as I described before, there is a noticeable delay from when the .torrent file is announced on IRC/RSS and available for download, and from when the tracker itself actually responds successfully to announce requests for that torrent.

I can speculate about the origin of the issue. When a torrent gets uploaded to a site, it goes to a directory and probably registered on some database. The associated tracker is probably polling a watch directory or database at a low enough frequency that allows this to happen.

You can easily try this out with TL or a similar private tracker; refer to some of my previous comments describing "racing". You will need an account on those trackers if you don't already have one, but I'm sure no private tracker will have a problem providing one to the main libtorrent dev.

[fusk-l]

I'm not done reading through all of this thread yet, but does "reconnect" refer to re-announcing to a tracker? If so, does anyone have a good understanding why trackers fail so frequently that this is an issue? are they simply overrun by traffic and don't have sufficient capacity?

It happens because a new torrent is made available before the tracker is updated which leads to the error.

[arvidn]

does anyone have a contact at a tracker that exhibits this problem? I think it would be interesting to know what their perspective is. Since they don't fix their trackers, presumably they expect clients to do something off-spec. Presumably they wouldn't want clients to hammer the tracker once a second until they get a response, so I wonder what they would think is the solution (assuming fixing the trackers isn't it).

[FranciscoPombal]

@arvidn

Since they don't fix their trackers, presumably they expect clients to do something off-spec.

Is it off-spec though? IIRC the bittorrent protocol spec just states that announces should be made both at regular intervals and when certain events occur, and optionally at the request of the client outside of the regular interval. Very fast regular intervals are still within the spec, even though for the common case it would be considered and abuse of the spec.

Presumably they wouldn't want clients to hammer the tracker once a second until they get a response, so I wonder what they would think is the solution (assuming fixing the trackers isn't it).

Hammering the trackers is already what happens. It is, in fact, the expected behaviour from clients who are racing.

I am not convinced that support for racing needs anything on libtorrent's side though. Presumably an option in qBit, upon adding the torrent "Hammer tracker within the first x minutes of adding torrent" would be enough? It's just force reannounces happening on repeat, every 1, 5, etc seconds until the tracker returns a valid response. See @fusk-l 's script above. It is a helper script for deluge to emulate this feature.

[arvidn]

Is it off-spec though?

yes. The tracker response includes an "interval" field, specifying how long the client should wait until re-announcing. The only legitimate reason to announce early is if the torrent is stopped or completes.

A later addition was the "min interval" field, to hint clients that they shouldn't let users force-announce within a certain minimum interval.

The original spec is a bit sparse and doesn't mandate any specific behavior if the tracker returns an error.

Hammering the trackers is already what happens. It is, in fact, the expected behaviour from clients who are racing.

It would be interesting to know what tracker admins would expect the "hammer-interval" to be. you can make connections every 1ms or every 5 seconds, or anywhere in between. As long as there is an incentive to "win the race", there's an incentive to hammer more frequently.

I am not convinced that support for racing needs anything on libtorrent's side though.

If you want the hammer interval to be short, libtorrent would have to do it probably.

Presumably an option in qBit, upon adding the torrent "Hammer tracker within the first x minutes of adding torrent" would be enough?

It's possible libtorrent would have to make some changes too, as I think there may be some enforcing of the "min interval" right now, preventing clients to force announce within it. But maybe this was tweaked a long time ago to allow Deluge to do this.

[FranciscoPombal]

@arvidn Thanks for the clarification on the spec. My brain froze and I completely forgot about the "interval" field logic. However, racers aren't supposed to keep hammering after a successful announce. After the successful announce, there is no racing incentive not to respect the interval field. Since the spec does not mandate any specific behavior if the tracker returns an error, as long as the clients only send very frequent announces until the error goes away, they are still within that gray area. But this is just nitpicking about what is off-spec and what's not at this point.

It would be interesting to know what tracker admins would expect the "hammer-interval" to be. you can make connections every 1ms or every 5 seconds, or anywhere in between. As long as there is an incentive to "win the race", there's an incentive to hammer more frequently.

True, but I don't think anyone is using <1 s intervals in practice. Then again, I am not a serious racer and I have no evidence to substantiate this claim. It is simply reasonable speculation I guess.

I guess you have to somehow contact private tracker staff (or they can try to contact you, if they see this thread) and ask them about what would be reasonable values. I would be happy to help with that, but I don't want to dox myself (I would like to keep this identity separate from the ones I use on private trackers). Like I stated before, I had some experience racing on TL in the past. My knowledge of racing comes from that.

If you want the hammer interval to be short, libtorrent would have to do it probably. It's possible libtorrent would have to make some changes too, (...)

Fair enough.

[fusk-l]

I don't think they care. Have the issue, sort it your self. I think the expected "hammering" is equal to how rutorrent does it, as it's probably the client that has had this feature as default for the longest. Don't remember the value but i think it's 5 secs.

[arvidn]

I think the main counter-weight / consideration is not so much the spec, but how easy it would be to take advantage of a popular torrent site to DDoS someone, by sticking them in the tracker field.

Is there a way to identify the kind of errors which would trigger this logic for instance? Presumably, failing by returning HTML should not allow continued hammering (even though some trackers do fail with HTML sometimes, when apache fails at a higher level for instance).

Presumably it would also be sufficient to enable this behaviour for torrents that are marked "private".

Anything else?

[FranciscoPombal]

@arvidn

I think the main counter-weight / consideration is not so much the spec, but how easy it would be to take advantage of a popular torrent site to DDoS someone, by sticking them in the tracker field.

Unlikely to happen, racing is only a thing on private trackers (and more-so on private "scene" trackers), which prevent that from happening, if I am not mistaken (i.e. they forbid the upload of torrents that have an incorrect tracker field). If that check is not in place in some tracker it is definitely their problem to fix.

Is there a way to identify the kind of errors which would trigger this logic for instance? Presumably, failing by returning HTML should not allow continued hammering (even though some trackers do fail with HTML sometimes, when apache fails at a higher level for instance).

At least TL returns something like "Torrent not found" in the message field, if I remember correctly. But, these messages are not a usable standard like HTTP status codes.

Presumably it would also be sufficient to enable this behaviour for torrents that are marked "private".

Yes, agreed.

Anything else?

I think that's all. @glassez , @fusk-l ?

[FranciscoPombal]

Also, it was already technically possible to DDoS someone by putting their address in the tracker field, given access to upload rights on a sufficiently popular public tracker. Just upload a torrent pretending to be the season finale of a well-known TV show, with a believable name, file names and size, near or at the expected release date. Never mind the frequency of announces, the sheer number of them would be a huge DDoS.

[arvidn]

Also, it was already technically possible to DDoS someone by putting their address in the tracker field, given access to upload rights on a sufficiently popular public tracker.

Yes, and this is the main reason I've been very hesitant to relax the logic I have in libtorrent that prevents frequent hammering. Including things like supporting HTTP 410 status code to mean "never talk to me again".

When it comes to DDoS, improving the effectiveness of it by 100x is significant and relevant.

[FranciscoPombal]

Yes, and this is the main reason I've been very hesitant to relax the logic I have in libtorrent that prevents frequent hammering. Including things like supporting HTTP 410 status code to mean "never talk to me again".

Supporting HTTP status codes would be a great thing on the client side. However, it will probably be a while before established trackers get updated to support that as well, limiting the effectiveness of the client-side support at first.

When it comes to DDoS, improving the effectiveness of it by 100x is significant and relevant.

I don't disagree with that. I am just saying that it's just unlikely to happen. DDoS'ing by injecting a victim address in the tracker field is not a thing now, in practice, and it won't become a thing with the introduction of these racing features, because the incentives to do such a thing are not there for tracker operators. Especially if those features are restricted to private torrents like you suggested, and they should be.

[arvidn]

I am just saying that it's just unlikely to happen. DDoS'ing by injecting a victim address in the tracker field is not a thing now, in practice

I don't think that's true. see this and this

[fusk-l]

Just want to point out that the ddos in the first link was caused by ignorance and a public torrent. On private trackers the rules are strict and something like adding unapproved urls is a no-go.

Especially if those features are restricted to private torrents like you suggested, and they should be.

If that's possible, this should not be an issue. Rutorrent is used be the majority of seedbox operators which run anywhere from 1 to 10gbit links per box. If you wanted to ddos those clients would be highly attractive, but so far i do not think there's been an incident like that as those boxes would typically run on private trackers.

[arvidn]

Just want to point out that the ddos in the first link was caused by ignorance and a public torrent. On private trackers the rules are strict and something like adding unapproved urls is a no-go.

I don't see what you're getting at. clearly a DDoS attack would not be launched with a private torrent on a private tracker. You would want maximum reach.

Especially if those features are restricted to private torrents like you suggested, and they should be.

If that's possible, this should not be an issue.

It's not obvious to me that it is. Presumably one can set the private flag on a torrent and still post it to a public tracker, or distribute it via other means.

Rutorrent is used be the majority of seedbox operators which run anywhere from 1 to 10gbit links per box. If you wanted to ddos those clients would be highly attractive, but so far i do not think there's been an incident like that as those boxes would typically run on private trackers.

In a DDoS attack based on tricking clients into announcing to a victim website, I don't think those would be particularly attractive. The rate of announces isn't materially different from any other random bittorrent client, presumably. You don't spend a significant amount of that bandwidth capacity announcing to trackers, is what I mean.

But talking about private trackers is missing the point a bit. We're talking about software that runs on a bunch of machines on the internet. That software need to avoid making it trivial to exploit it to launch attacks. The fact that some people only use the software with certain services isn't really relevant.

[arvidn]

I think, in an ideal world, trackers would opt-in to be hammered. Perhaps with some mechanism like this http://bittorrent.org/beps/bep_0034.html

[arvidn]

what's the current work-flow to add a torrent from an IRC channel? is there a separate tool that talks to qBT over a webUI connection to add the torrents? if so, it may be sufficient to have an opt-in check-box in the API call to add a torrent to enable this behaviour. That way there wouldn't be any risk of a random user accidentally enabling it.

An interesting question, though, is whether the torrents posted in the IRC channel are authenticated somehow, or can anyone start posting tons of torrents in there? (that aren't necessarily approved by the private tracker)

[FranciscoPombal]

@arvidn

I don't think that's true. see this and this

It's not obvious to me that it is. Presumably one can set the private flag on a torrent and still post it to a public tracker, or distribute it via other means.

But talking about private trackers is missing the point a bit. We're talking about software that runs on a bunch of machines on the internet. That software need to avoid making it trivial to exploit it to launch attacks. The fact that some people only use the software with certain services isn't really relevant.

These are fine points, but in my comment above, I said that the attacks don't really happen in practice. The first link was an honest mistake, the second one is a theoretical exploration of the possibility. In the real world, at least to my knowledge, this has never happened at any sort of large scale with malicious intent. After all:

• public trackers and public torrent indexers have no incentive to do this since it would tarnish their reputation and thus hurt their ad revenue due to decreased visits eventually. DDoS'ing with this method brings no obvious benefit to them, at a potentially great cost.
• private trackers and indexers don't do this for obvious reasons.

Regardless of what the actual prevalence of this attacks is, I am trying to argue that adding these new features would not change it significantly, if the implementation is done the right way, like what you said here:

it may be sufficient to have an opt-in check-box in the API call to add a torrent to enable this behaviour. That way there wouldn't be any risk of a random user accidentally enabling it.

This is what I was trying to get at. Sorry if I did not make it clear from the beginning. Just something on the advanced settings, disabled by default, that racers will want to toggle, but that the vast majority of regular users will leave as is.

what's the current work-flow to add a torrent from an IRC channel? is there a separate tool that talks to qBT over a webUI connection to add the torrents?

The usual workflow for qBT or most other clients is to have something like https://github.com/autodl-community/autodl-irssi listening in on IRC announce channels for the desired torrents and download them into a directory watched by qBT. No special webUI features are used, just the watched folder feature.

An interesting question, though, is whether the torrents posted in the IRC channel are authenticated somehow, or can anyone start posting tons of torrents in there? (that aren't necessarily approved by the private tracker)

Private tracker IRC announce channels are restricted to members of said private trackers. Furthermore, only users with upload rights can upload torrents that then get announced on the IRC announce channels. The torrent files are either sanitized first to ensure they only contain the private tracker's announce URL and, if they are not, some rogue user that decides to put extra addresses won't go very far before they get banned. On the IRC channel, the torrents are actually announced by bots that watch the website/database for new files

[fusk-l]

Just want to point out that the ddos in the first link was caused by ignorance and a public torrent. On private trackers the rules are strict and something like adding unapproved urls is a no-go.

I don't see what you're getting at. clearly a DDoS attack would not be launched with a private torrent on a private tracker. You would want maximum reach.

Especially if those features are restricted to private torrents like you suggested, and they should be.

If that's possible, this should not be an issue.

It's not obvious to me that it is. Presumably one can set the private flag on a torrent and still post it to a public tracker, or distribute it via other means.

Rutorrent is used be the majority of seedbox operators which run anywhere from 1 to 10gbit links per box. If you wanted to ddos those clients would be highly attractive, but so far i do not think there's been an incident like that as those boxes would typically run on private trackers.

In a DDoS attack based on tricking clients into announcing to a victim website, I don't think those would be particularly attractive. The rate of announces isn't materially different from any other random bittorrent client, presumably. You don't spend a significant amount of that bandwidth capacity announcing to trackers, is what I mean.

But talking about private trackers is missing the point a bit. We're talking about software that runs on a bunch of machines on the internet. That software need to avoid making it trivial to exploit it to launch attacks. The fact that some people only use the software with certain services isn't really relevant.

I see, thanks for the correction.

As mentioned above autodl-irrsi is typically the go-to tool, it's more or less the only tool an no alternative really exists. Autodl contains lists for basically every private trackers announce channel, does the filtering and downloads torrent to a watchfolder for the client to snatch.

[arvidn]

I'm leaning towards this being a client-level feature too though, and not have built-in support in libtorrent (until someone can demonstrate a clear benefit of some knowledge or ability that's only available at the libtorrent level)

[FranciscoPombal]

@arvidn

does anyone have a contact at a tracker that exhibits this problem? I think it would be interesting to know what their perspective is.

If you are still interested in joining a tracker where you can properly test all things racing related, this one has opened sign ups for the holidays (use the code in the reddit post).

https://www.reddit.com/r/OpenSignups/comments/e8ou3r/torrentleech_opens_signups_code_tlfreeleech/

If you get into trouble with ratio while testing, try contacting staff, they might open some exceptions since you're the main libtorrent developer.

[FranciscoPombal]

Renamed the issue to reflect the fact that we are basically discussing racing here. The part about the libtorrent configuration can be discussed here: https://github.com/qbittorrent/qBittorrent/issues/7585

[fleusium]

This is not necessarily connected to "racing" but I think it would be nice to have the option to lower the announce interval to around 1-5 minutes. It would allow the client to react more dynamically to the ever-changing seed/peer availability.

Imagine that you're downloading a torrent with only 1 seeder and he's only available for 1 hour per day. With the current 30 minute announce interval you're losing up to 50% of that precious download time. For very slow torrents that could mean the difference of weeks or even more in download time.

[arvidn]

@vikusz the announce interval is decided by the tracker. I would expect most trackers set it based on their load, or at least to avoid being overloaded or receive an unreasonable amount of traffic.

I think it's somewhat hostile for a client to announce more frequently than a tracker is asking for. Although, clearly clients do sometimes (perhaps by mistake). If torrent is stopped and immediately started again for instance. Some clients have special logic to announce frequently if it doesn't get any peers from the tracker (and for this reason there's an informal convention of trackers also specifying min-interval, the shortest possible re-announce interval the client should make).

Anyway, if a tracker has the capacity and thinks it benefits the swarm to have shorter announce intervals, it can shorten it.

Regarding your example; there are a few important assumptions you're making. I think it's these ones:

1. it's a private torrent and you can't find peers via any other means than the tracker (like, PEX or DHT)
2. You are bihind a NAT and have failed to open up ports to make your client connectable from the outside.

If one of those isn't true, I don't think you would have the outcome you suggest.

[fleusium]

@arvidn I didn't realize that's the case. It makes sense. Thanks for the explanation.

[bob454522]

I appreciate all the replies from the dev in this thread (or someone close to the dev), but FWIW, im seeing this same issue.

ill have rtorrent (rutorrent) on one VM and qbit on another (win 2012r2) , and qbit will be the only one stuck on "stalled" (bc of tracker "Not Working"), if i click it and choose force re-announce, it then starts working/downloading.

[ckcr4lyf]

From what I've tested, "reannounce" does not actually reannounce to the tracker, if I try it before the min interval expires. (Tested by running a custom tracker and viewing HTTP request logs)

The reason it is useful, is for some reason, private trackers make the .torrent available before it is in their tracker/announce database. As a result, when you announce, the hash is returned as invalid. But since its HTTP 200, qbit respects the min interval.

Often for some sites, the only way to bypass this is by manually reannouncing every 5 seconds until the announce is OK. This is an "accepted" practice for "racing".

One alternative is to pause and resume the torrent. This works, but if this is possible to "re-announce" regardless of min interval, it would be nice if we could manually reannounce even within the min interval.

In terms of "spamming" the tracker, I think if trackers do have a problem they can implement soft/hard bans on their end. But an explicit "reannounce" option withing qBit should bypass the min interval.

[YourHuckleberry]

I've been experiencing this myself (with TL, specifically). I don't think this is a problem for qbit/libtorrent to solve. I also experienced this with deluge, and found a script that I ran on the addition of a new torrent that polled every two seconds and paused/resumed the torrent until it got a valid torrent response. Not elegant, but lazy and effective.

The trackers shouldn't be announcing if there's no torrent available, but I believe that the site race encourages them to announce as soon as they have details even though the torrents seeder may not have actually connected yet.

There are two utilities that I am aware of that allow irc downloading - autodl-irssi and a newer option called trackarr. Asking all torrent sites to fix this won't work, obviously, so to me the path of least resistance seems to be to adjust the irc bots themselves (of which there are only two I know of) to check the torrent before sending it on to it's destination.

[ckcr4lyf]

This is a measure taken by qbittorrent to "prevent abuse" by reannouncing too much - personally I find this a little weird, since a malicious actor could just curl an http tracker or write a script for a UDP one.

Either way, I created a program called qbit-race - https://github.com/ckcr4lyf/qbit-race , to solve this problem. For trackers where they upload .torrent before its in their announce DB, it will keep reannouncing (rather than pause-resume), until it announces correctly. Compatible with autodl-irssi

[arvidn]

This is a measure taken by qbittorrent to "prevent abuse" by reannouncing too much - personally I find this a little weird, since a malicious actor could just curl an http tracker or write a script for a UDP one.

I may not understand what you mean, but it sounds like you're saying: since it's possible to be abusive, qBT may as well be.

[ckcr4lyf]

Sorry, I didn't word it in the best possible way. It's more like, it is not possible to force a reannounce, if a successful one has been made in within the last min_interval. And actually this is fine for 99.83% purposes (including racing) - when a torrent is unregistered, the announce fails, and as such it is indeed possible to announce again instantly.

I think this was some confusion a lot of people had initially - but it is indeed possible to reannounce while it is still unregistered.

I apologize for the poor wording in my previous comment.

[fusk-l]

This is a measure taken by qbittorrent to "prevent abuse" by reannouncing too much - personally I find this a little weird, since a malicious actor could just curl an http tracker or write a script for a UDP one.

Either way, I created a program called qbit-race - https://github.com/ckcr4lyf/qbit-race , to solve this problem. For trackers where they upload .torrent before its in their announce DB, it will keep reannouncing (rather than pause-resume), until it announces correctly. Compatible with autodl-irssi

That's awesome, i have literally been using deluge the past few years on my box, simply because qbit couldn't do this. Deluge could with a plugin i help make for it. But i've always wanted to go back to qbit because deluge doesn't run very well. Hasn't been updated in years and is now maintained by forum users, but it worked. This might be what i need to get back to qbit.

Edit: Just had a look at it, i have no idea how to use that.

[klunky]

There is no way to reset the min_interval parameter? The original Deluge script posted on the beginning on the thread does exactly that, and that's why Deluge is the preferred client to race

[fusk-l]

I don't know what the original script does as i never got that to work. But the plugin created afterwards looks at the "tracker status" and continues to re-announce until "Announce OK" is detected. That works.

[ckcr4lyf]

There is no way to reset the min_interval parameter? The original Deluge script posted on the beginning on the thread does exactly that, and that's why Deluge is the preferred client to race

No, because this is decided by the tracker. You should not send an announce before min_interval expires, if you have had a successful announce already