high/realtime priority packet spam

[GoogleCodeExporter]

Reported by qbt user here: 
https://github.com/qbittorrent/qBittorrent/issues/1877

He uses libtorrent 0.16.17. I copy/paste his report below.

qBittorrent is spamming high priority packets at times, causing it's own 
up/download to stall at 0kb/s. All other clients in my network are also unable 
to do anything and lose connection. The router shows that some application 
requested realtime priority for the packets and my whole bandwidth is used by 
it.

Execution log does not show anything related. The only way to unclog the 
connection is to

    reconnect using the router webUI
    turn off qbittorrent

What is it uploading at realtime priority and how do I turn it off?

This time it seems to be related to a new download starting from the RSS 
downloader, however after restarting qb once to stop its spam, it started doing 
it again without the RSS downloader triggering while I typed out the first 
post, eating it in the process.

Original issue reported on code.google.com by hammered...@gmail.com on 14 Aug 2014 at 7:33

[GoogleCodeExporter]

what does "realtime priority" and "high priority" mean? is that a dscp?

how was it determined that these packets caused stalls?
If some libtorrent packets are being sent at a rate that starves out other 
traffic, why would the upload rate be 0 kB/s?

does qbittorrent modify the TOS setting?
could you provide a packet capture of these packets?

Original comment by arvid.no...@gmail.com on 14 Aug 2014 at 11:57

[GoogleCodeExporter]

>does qbittorrent modify the TOS setting?

Searching for 'peer_tos' in the qbt source didn't yield anything.

As for the other questions, I'll wait for the user to comment.

Original comment by hammered...@gmail.com on 15 Aug 2014 at 12:11

[GoogleCodeExporter]

I'm the user who reported it.

The priorities is how my router webUI shows them, here is an example how
the traffic looks like:
https://dl.dropboxusercontent.com/u/19330332/weird_issue.png

The update rate of the router webUI is about 2-3 seconds. There are four
levels: realtime, high, normal and background.
I can setup rules by defining the ports and then apply these to clients,
identified by their IPs. There are no realtime rules defined except for
VoIP, so it's not the router that modifies the priority.

My home network consists of several PCs connected via LAN as well as
several mobile devices, over WLAN. One of the PCs is a headerless linux
server maintained over VNC.
To narrow down what was causing the spam, I did the following:
- wait until connection became seemingly dead
- verified via router webUI that the connection is present
- noticed my upload bandwidth was full with realtime packets
- turned off WLAN (rules out mobile devices)
- unplugged PC one by one, checking if the spam stopped
- only two PC were left, the one running qBittorrent and mine
- qB did not show any traffic up/down
- KSysGuard showed some upload traffic on the server, caused by the vnc
connection+some delta (more than ~70kb/s, my max upload)
- Task manager on my PC showed >70kb/s down (vnc) and <70kb/s up (so it's
not my PC using all of it)
- turned off qBittorrent
-> realtime traffic stops.

As for why up/down rate would be 0kb/s, I can only speculate it might be a
traffic that is not reported on the qB GUI.
I never had to capture packets before, so you'd have to walk me through
(linux). Also, the behavior only shows up from time to time, so it might
take a while until I'm able to see it.
First time it took about 4 days to happen, yesterday it happened within two
minutes. It also never happened before I switched to qB from µTorrent
(wine) a week ago.

Original comment by chrnosph...@gmail.com on 15 Aug 2014 at 11:18

[GoogleCodeExporter]

Can you try with disabled dht/lsd/pex? Maybe these cause excessive traffic to 
maintain nodes.

About capturing packets, I think he wants you to use wireshark. Install it on 
the headless machine and have it record traffic going through the network 
interface. (close all other apps that use the network on that machine). You may 
be able to define filters to cut down the captured traffic, but I don't know 
which. arvid might help on this.
Then you should email him the captured data and not post it here directly. (I 
assume that they might contain login passwords). 

Original comment by hammered...@gmail.com on 15 Aug 2014 at 12:09

[GoogleCodeExporter]

One Scenario I have seen which creates network traffic which is not reported by 
libtorrent is caused by an utp problem. Libtorrent enters an infinite 
acknowledge loop with a peer and it seems that the higher level in libtorrent 
have already lost track of that peer. In my case I got upload/download rates up 
to ~40kbytes/s.
See if disabling utp solves the problem.

Original comment by webmas...@massaroddel.de on 16 Aug 2014 at 4:16

[GoogleCodeExporter]

if anyone can reproduce such uTP issue, please build with utp logging enabled 
(define TORRENT_UTP_LOG to 1 in src/utp_stream.cpp and maybe also 
TORRENT_VERBOSE_UTP_LOG)

Original comment by ar...@bittorrent.com on 16 Aug 2014 at 5:35

[GoogleCodeExporter]

in general though, to help troubleshoot this issue, the following would be 
useful:

1. find out what your router means by "high priority traffic". Is it just 
labeling all UDP traffic as high priority?

2. disable features to find if the problem appear to be caused by a specific 
feature. things like, local peer discovery, uTP, DHT for instance.

3. when you experience this problem, capture a minute or so of traffic with 
wireshark. The capture may contain sensitive/private information, but is not 
very likely to (unless you use old school protocols like FTP, telnet or HTTP 
for logins)

Original comment by arvid.no...@gmail.com on 16 Aug 2014 at 5:41

[GoogleCodeExporter]

I have downgraded to qBittorrent 3.1.9.2 and the issue has not resurfaced so 
far. I will report back should it happen again. If it doesn't, though, then 
it's more likely it's a qB bug and not a libtorrent one, since the libtorrent 
version didn't change...

Original comment by chrnosph...@gmail.com on 16 Aug 2014 at 9:08

[GoogleCodeExporter]

I don't think there is a difference in what 3.1.9.2 and 3.2.0alpha does in 
setting up libtorrent. I think the settings are identical.

Original comment by hammered...@gmail.com on 16 Aug 2014 at 9:14

[GoogleCodeExporter]

I'm just speculating at the moment.
I also managed to install Wireshark, so i should be able to capture the
packets should the issue reappear.

Original comment by chrnosph...@gmail.com on 16 Aug 2014 at 9:29

[GoogleCodeExporter]

#5 might be spot on. According to the stuff I see on wireshark right now, it is 
a spam of ACK/PSH,ACK packets. This time it only lasted for about a minute 
before recovering, but was enough to cause a "disconnect" for all the other 
programs. I will try to disable utp and see if that fixes it. In the meantime, 
here are the captured packets: 
https://dl.dropboxusercontent.com/u/19330332/utp_packet_spam.7z

Original comment by chrnosph...@gmail.com on 17 Aug 2014 at 12:20

[GoogleCodeExporter]

Update: it's not utp, or DHT (local/peer exchange), and I don't have an idea 
what it might be. Here is one more capture, with utp/dht disabled and me 
shutting off qBittorrent at around 35 seconds in:

https://dl.dropboxusercontent.com/u/19330332/utp_packet_spam2.7z

Original comment by chrnosph...@gmail.com on 17 Aug 2014 at 4:19

[GoogleCodeExporter]

In that dump, most of the packets are primarily in two streams. one is 
classified as VNC (I'm not sure it actually is though, wireshark sometimes has 
pretty unsophisticated heuristics), the other one is a TCP connection that also 
isn't obvious what it is. The VNC flow does not look like anything bittorrent 
related though. The TCP connection _could_ be bittorrent, it's hard to say. 
It's uploading at about 100kB/s during the first half of the capture, then it 
drops to about 25 kB/s.

there is a surprising amount of incoming duplicate acks regularly. This 
connection is then reset by the local computer at the end of the capture (this 
is presumably when you shut down qBT).

There are about 20 outgoing TCP connection attempts per second, until the drop 
(at 35 seconds in). From then on there are no outgoing connection attempts.

Did you shut down qBT at the end or in the middle of the capture?

Original comment by arvid.no...@gmail.com on 17 Aug 2014 at 8:53

[GoogleCodeExporter]

As you have guessed,  I have shut it off at the 35sec mark for the second
one. For the first one,  it somehow recovered by itself near the end of the
capture.

Original comment by chrnosph...@gmail.com on 17 Aug 2014 at 8:57

[GoogleCodeExporter]

actually, the TCP connection is reset 36 seconds in. I'm guessing that's when 
you closed qBT.

Is it possible that your router classifies traffic based on source or 
destination port, and prioritizes based on that? if so, that would explain why 
a random TCP connection (presumably a bittorrent connection seeding) would 
starve everything else out.

Original comment by arvid.no...@gmail.com on 17 Aug 2014 at 9:18

[GoogleCodeExporter]

I don't have any such rules defined though. The only rule that _might_ fit
is one that automatically sets all outgoing traffic on ports 1024 and up to
lowest priority.
However, I do not think it's the router's fault, as all other torrent
clients I used were configured using the same ports. I have this rule for
about 3 years now and such a behavior never popped up with other torrent
clients (namely, azureus, µtorrent, ktorrent). I also switched to
qBittorrent for a short while last year (3.1.8.x I think), and the issue
wasn't present there either. So it must have been introduced somewhere
after that...

Could optimization options for gcc have broken something? I am using the
arch linux recommended makepkg config, which are:

CFLAGS="-march=native -O2 -pipe -fstack-protector --param=ssp-buffer-size=4
-D_FORTIFY_SOURCE=2"
CXXFLAGS="${CFLAGS}"

I now have gone back to µtorrent in the mean time and am going to monitor
for it to happen again, but I really doubt it to be honest.

Original comment by chrnosph...@gmail.com on 17 Aug 2014 at 9:34

[GoogleCodeExporter]

The behavior hasn't reappeared since then, so I'm concluding it must be
something about libtorrent. Is there any more info I can provide to get
this fixed? Running µtorrent in wine has it's own share of problems it
seems and I'd rather use a native libtorrent-based client.

Original comment by chrnosph...@gmail.com on 19 Aug 2014 at 8:47

[GoogleCodeExporter]

I had another look at the suspect TCP flow, specifically the DSCP field, but it 
all looked normal. Do you think you could find out what your router means by 
"realtime priority"? perhaps there's a manual or a support forum where people 
may know.

That would certainly help in knowing what to look for.

Original comment by arvid.no...@gmail.com on 19 Aug 2014 at 9:19

[GoogleCodeExporter]

I will send AVM (the company) a mail asking for that info. By the way, I
always forget to mention it, but the reason why the traffic is not
identified as bittorrent is probably because I am using (forced) encryption.
Would it be useful to try and reproduce the issue with encryption turned
off?

Original comment by chrnosph...@gmail.com on 19 Aug 2014 at 9:35

[GoogleCodeExporter]

yeah, that seems like a reasonable test.

Original comment by arvid.no...@gmail.com on 19 Aug 2014 at 10:09

[GoogleCodeExporter]

Got an answer from the support. They don't really have any detailed 
information, but one thing they could tell me is that TCP ACK is always sent 
with high priority.

Which got me thinking, I was relying on the router's QoS rules to manage the 
upload, meaning qBittorrent was running at unlimied upload. Could it be that 
this somehow leads to ACKs getting "stalled", then sent in a burst, "clogging" 
the connection? Though if they are always high priority, why do they get 
delayed anyway.

In any case, I have limited the upload to slightly below my maximum capacity 
and so far it did not reappear, so I am going to continue monitoring it for now.

Original comment by chrnosph...@gmail.com on 20 Aug 2014 at 12:18

[GoogleCodeExporter]

ok. if you're up for it, it might be interesting to reproduce and capture the 
problem without protocol encryption. I wonder if there's anything at the 
bittorrent protocol level that might cause TCP to flip out.

It's not obvious what it means thought, that "TCP ACK is always sent with high 
priority". Essentially all TCP packets (except for the initial handshake) are 
ACKs. they have the ACK flag set and the ACK sequence number is relevant.

Original comment by arvid.no...@gmail.com on 20 Aug 2014 at 4:30

[GoogleCodeExporter]

I am up for it. I'm guessing capturing unencrypted packets will show what 
exactly I'm downloading? 

I've been running qB with DHT off an limited upload for over a day now and so 
far no re-occurrence. I will enable DHT tomorrow and see it if re-introduces 
the issue. Otherwise, it might have something to do with running libtorrent 
with no upload limit set... idk.

Sadly, they won't be able (or willing) to provide any more info it seems. I'm a 
bit rusty when it comes to network protocols, but wasn't there an ACK sent to 
the peer for every data packet you receive? Isn't data also sent via TCP? Or is 
it UDP in this case?

Original comment by chrnosph...@gmail.com on 21 Aug 2014 at 8:25

[GoogleCodeExporter]

Been running without issues for more than a week now. 
Moral of the story seems to be: don't rely on router QoS, limit your upload 
according to your upstream.

If you want to find out why libtorrent/my router might be freaking out when no 
upload limit is set, I'm willing to try out unlimiting it again and capturing 
some packets.

Original comment by chrnosph...@gmail.com on 31 Aug 2014 at 8:30

[GoogleCodeExporter]

In the end, it wasn't the limiting. I'm seeing the following in my journal:

kernel: TCP: request_sock_TCP: Possible SYN flooding on port 12120. Sending 
cookies. Check SNMP counters.

etc. I have tried playing around with the kernel values for net.ipv4, for 
example raising or lowering the synack queues, but that doesn't seem to affect 
anything. I've also tried to disable syncookies, but that doesn't do anything 
either.

The change I'm having now with libtorrent 1.0.3.0 and a relatively fresh 3.2.0 
git snapshot of libtorrent is that qB actually shows traffic now - it's 
uploading at maximal capacity. I can also see peers disconnect one by one until 
the one causing is left. This time it was some Korean IP connected via the BT 
protocol using µtorrent 3.4.2 After banning it via qB, traffic instantly 
returned to normal.

Original comment by chrnosph...@gmail.com on 21 Feb 2015 at 6:22

[GoogleCodeExporter]

Sorry, meant to write "and a relatively fresh 3.2.0 git snapshot of 
qBittorrent", not libtorrent

Original comment by chrnosph...@gmail.com on 21 Feb 2015 at 6:25

[GoogleCodeExporter]

do you think that peer is malicious? it almost sound like some kind of DoS 
attack. if so, i would be interested in a wireshark capture of it, including 
you banning the peer and restoring your rates.

can you see in the qB logs why the other peers dropped off?

Original comment by arvid.no...@gmail.com on 21 Feb 2015 at 8:16

[GoogleCodeExporter]

I'm not sure what to think to be honest. Other peers with the same µtorrent
version which are also using the BT protocol don't seem to cause any
issues, but that's the first time I noticed it was a specific peer that is
causing that. So, maybe the peer is malicious. The execution log doesn't
seem to show anything regarding the peers dropping off, the only thing
captured is me manually banning the peer in question.

I'll try unblocking the peer and capturing the traffic if that happens
again.

Original comment by chrnosph...@gmail.com on 21 Feb 2015 at 8:56

[GoogleCodeExporter]

I managed to catch another single peer that caused this. I don't think it's a 
malicious peer, the country and IPs are all over the globe.

Here is the traffic of the overload, with me banning the IP at the end:
https://dl.dropboxusercontent.com/u/19330332/packets_single_ip.7z

Alternatively, of here are the packets of all ips at that point (filtered out 
local traffic):
https://dl.dropboxusercontent.com/u/19330332/packets_all_ip.7z

Hope that can help somehow.

Original comment by chrnosph...@gmail.com on 3 Mar 2015 at 12:43

[GoogleCodeExporter]

Here's another one, different IP again
https://dl.dropboxusercontent.com/u/19330332/packets_all_ip2.7z

Same scenario, only one peer active, upload is full, banning the peer makes 
other peers return.

Original comment by chrnosph...@gmail.com on 5 Mar 2015 at 4:29

[GoogleCodeExporter]

Don't know if it's important, this is the state of the peer according to 
qBittorrent before I banned it:

Connection: BT
Flags: D U I E
Client: µTorrent 3.4.2
Down Speed: 20.8KiB/s
Up Speed: 63.2KiB/s 

My max is 700KiB/s down and 70KiB/s up, with limits set to 400KiB/s and 68KiB/s 
respectively. The limits are also set to apply them to protocol overhead.

Original comment by chrnosph...@gmail.com on 5 Mar 2015 at 4:34

[GoogleCodeExporter]

Flag explanation:
D = "interested(local) and unchoked(peer)"
U = "interested(peer) and unchoked(local)"
I = "incoming connection"
E = "encrypted traffic"

Relevant qbt code source: 
https://github.com/qbittorrent/qBittorrent/blob/19b9a845761450e6ce9ca185498ddde3
b696aa95/src/gui/properties/peerlistwidget.cpp#L479

Original comment by hammered...@gmail.com on 5 Mar 2015 at 5:15