search

arvidn / libtorrent

an efficient feature complete C++ bittorrent implementation
http://libtorrent.org
Other
5.23k stars 995 forks source link

Duplicate statistics on trackers #7324

Open Aleksman4o opened 1 year ago

Aleksman4o commented 1 year ago

libtorrent version (or branch): 1.2 from master

platform/architecture: linux, qBittorrent 4.6.0/x64

When you have 2 or more internet connections and you have set up a listening port on both of them, announcements to the tracker go through both of them and this is the correct behavior. But the statistics on the tracker are doubled, which may be undesirable in some cases. Now I have 2 Internet connections and I solved this problem for myself by simply dividing the statistics given to the tracker by 2. Maybe this can be done automaticcaly by dividing this statistics into the number of listened interfaces? photo_2023-02-23_02-29-11

Seeker2 commented 1 year ago

Possibly related to: Massive duplicate announces by libtorrent clients ...or part of the causes of that anyway.

Aleksman4o commented 1 year ago

Possibly related to: Massive duplicate announces by libtorrent clients ...or part of the causes of that anyway.

Don't think so. In my case announcements provide correct statistics with 1 internet connection. But it is multiplied by the number of internet connections. Therefore, it should be divided by the number of interfaces, or, alternatively: statistics should be sent through the first interface, zeroes should be sent through others.

rafi-d commented 1 year ago

Unlike you, I thing that the mistake here is not the stats, but sending duplicate announce requests on more than one connection.... Next, you'll ask for two port numbers, and two internal outgoing ports, plus load balance the seeding between the two..

Everything is possible, but it's just too complicated, and prune to more bugs, IMHO...

Aleksman4o commented 1 year ago

But it's possible already and works fine. I can get full speed of both connection, just a bug with statistics, which does not seem difficult to fix

arvidn commented 1 year ago

Good discovery. I don't think the solution is obvious. The issues happen when interfaces go down and come back up again.

Or when announces fail.

By default, libtorrent will attempt all interfaces, even ones that will never succeed (like loopback, or perhaps a dead VPN interface). It's not trivial to know what to divide by.

I think an ideal solution would be to track statistics separately by the interface, but that gets pretty complicated. I'm not sure that's worth it.

Aleksman4o commented 1 year ago

May be my solution not ideal, but at start we can see detected external IP's and we can divide by it's number.

rafi-d commented 1 year ago

Why is it that wrong to count all the messages on all interfaces (and not devide by x)?

Aleksman4o commented 1 year ago

Because you can receive ban on trackers for cheating and just to see the right statistics.

Roardom commented 1 year ago

As a tracker, we dug into this issue back in ~2021 and had to implement a workaround where we invalidate all further announce-triggered database updates from a user per torrent for the next 30 seconds. 10gig dual stack seedboxes running qbittorrent would often have 4 announces due to them having both a 10g and 1g nic, even though the tracker itself doesn't have an AAAA record. We also noticed an issue with the "announce to all tracker tiers" option which was causing duplicate announces when enabled (on private torrents with only a single tracker added), even on one 1g dual stack nic (causing 3 announces). It's been awhile since we dug into this so it's not as fresh in my mind as it used to be, so I won't be of too much help, but figured I'd note down what I do remember just in case.

Roardom commented 1 year ago

I just noticed I misread the issue. Are you saying that if you upload e.g. 10 bytes, that you announce to the tracker twice, once from each interface, each with an uploaded value of 10 bytes? Or that you announce with an uploaded value of 20 bytes? I'm only seeing the first case, which most private trackers (ocelot/radiance/unit3d/nexusphp) are already capable of handing as they take the delta between announces.

Aleksman4o commented 1 year ago

I just noticed I misread the issue. Are you saying that if you upload e.g. 10 bytes, that you announce to the tracker twice, once from each interface, each with an uploaded value of 10 bytes? Or that you announce with an uploaded value of 20 bytes? I'm only seeing the first case, which most private trackers (ocelot/radiance/unit3d/nexusphp) are already capable of handing as they take the delta between announces.

first

Roardom commented 1 year ago

In that case, if you modified the code to halve your uploaded and downloaded, then the tracker would only receive half your total values, since it would only increase your onsite upload/download by the difference in between your past 2 announces. On the other hand, setting every second interface to 0 would cause every second announce to increase your onsite upload/download by the values of the interface that isn't set to 0, constantly increasing for however long you seed it. Both methods would be considered as cheating and would get your account flagged. If you were seeing duplicate values on the tracker prior to your change, they probably have a race condition in their code.

Seeker2 commented 1 year ago

10gig dual stack seedboxes running qbittorrent would often have 4 announces due to them having both a 10g and 1g nic, even though the tracker itself doesn't have an AAAA record.

Just to clarify... Does this mean there are additional tracker announces for every NIC and/or interface (1 for IPv4 and 1 for IPv6) even if these are not internet-facing? (such as a LAN ip address with no internet gateway)

arvidn commented 1 year ago

Does this mean there are additional tracker announces for every NIC and/or interface (1 for IPv4 and 1 for IPv6) even if these are not internet-facing? (such as a LAN ip address with no internet gateway)

Not intentionally at least. But I'm interested if anyone sees evidence of this

Roardom commented 1 year ago

When I contacted the seedbox provider about this issue (they hardcode the libtorrent configuration settings when building qbittorrent), they did some testing, and told me that when the announce to all tracker tiers setting is enabled, if the machine has a valid IPv6 (The local IPv6 that Debian has by default doesn't trigger it), then it will trigger a duplicate announce. They now disable this setting in all their builds going forward.

The issue with announce to all tracker tiers is separate from clients sending duplicate announces. If a client has multiple interfaces selected on a dual stack machine, then it will announce from each interface. And if the seedbox is dual stack and the tracker is only IPv4 (no AAAA record), the tracker will still receive 2 announces (originating from the same IPv4, exact same query parameters, milliseconds apart).

I dug back into my notes from 2 years ago, and noted that combined, these 2 bugs actually only combine to a total of 3 announces. The scenario where we received 4 announces seems to be unique to only a couple of racers at the time, likely with 20gig machines (or 2x10g, not sure how that would work).

ip a output of affected 3-announce machine: https://pastebin.com/QMGNbYTi

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Seeker2 commented 1 year ago

This might be easier to resolve if libtorrent didn't report different peerIDs on each interface it uses and even different from the ones it sends to trackers.

stalkerok commented 1 year ago

HDBits - 2023-08-21

qBitTorrent and counting download/upload multiple times If you use qBitTorrent and have several interfaces open to the internet, make sure to disable multi-homed feature in the qBitTorrent, set preferred interface so it doesn't announce on all of them. This is a problematic feature, it's basically cheating, it sends the same data several times, tracker receives it in parallel and processes it and adds the amount multiple times to download and upload. Go to Options - Advanced - Network interface (3rd option) and change the value "any interface" to your primary interface (only needed if you have several active connections like LAN and Wifi at the same time).

ghost commented 1 year ago

This might be easier to resolve if libtorrent didn't report different peerIDs on each interface it uses and even different from the ones it sends to trackers.

The peer ID sent to peers is unrelated to the statistics reported to the tracker. The peer ID remains constant for all interfaces when communicating with trackers. These are distinct matters and should not be confused with each other.