Too many listening sockets with default settings

[aldenml]

libtorrent version: master in macOS

When starting a session with the default settings, I get to many listening sockets:

successfully listening on [TCP] 0.0.0.0:6881
successfully listening on [UDP] 0.0.0.0:6881
successfully listening on [TCP] [fe80::1%lo0]:6881
successfully listening on [UDP] [fe80::1%lo0]:6881
successfully listening on [TCP] [fe80::18a7:e069:bf6:3432%en0]:6881
successfully listening on [UDP] [fe80::18a7:e069:bf6:3432%en0]:6881
successfully listening on [TCP] [2602:30a:2e11:e60:1490:eadc:4fef:7db0]:6881
successfully listening on [UDP] [2602:30a:2e11:e60:1490:eadc:4fef:7db0]:6881
successfully listening on [TCP] [2602:30a:2e11:e60:6878:c590:1dc3:ea83]:6881
successfully listening on [UDP] [2602:30a:2e11:e60:6878:c590:1dc3:ea83]:6881
successfully listening on [TCP] [fe80::1c37:b9ff:fe98:93f5%awdl0]:6881
successfully listening on [UDP] [fe80::1c37:b9ff:fe98:93f5%awdl0]:6881
successfully listening on [TCP] [fe80::a5fc:4b85:2635:18fe%utun0]:6881
successfully listening on [UDP] [fe80::a5fc:4b85:2635:18fe%utun0]:6881
successfully listening on [TCP] [fe80::896f:5ab0:708f:ae9%utun1]:6881
successfully listening on [UDP] [fe80::896f:5ab0:708f:ae9%utun1]:6881
successfully listening on [TCP] [fe80::429:7efd:4213:8765%en4]:6881
successfully listening on [UDP] [fe80::429:7efd:4213:8765%en4]:6881
successfully listening on [TCP] [2602:30a:2e11:e60:c71:5300:8558:5d34]:6881
successfully listening on [UDP] [2602:30a:2e11:e60:c71:5300:8558:5d34]:6881
successfully listening on [TCP] [2602:30a:2e11:e60:95ea:a74d:c0ca:e7be]:6881
successfully listening on [UDP] [2602:30a:2e11:e60:95ea:a74d:c0ca:e7be]:6881

I assume some of them could be filtered out easily, but I want to have a more technical criteria other than just filter by obvious name.

One particular question, is the opening of UDP in IPv6 [fe80::1%lo0] right?

[ssiloti]

Do you mean having libtorrent reduce the set of sockets it listens on by default or just filtering for display? For the former, libtorrent shouldn't be second guessing which interfaces the user wants to listen on. In most cases listening on more interfaces than necessary is harmless. If the user cares, they can use the listen_interfaces setting to refine the set of listen sockets. For the latter it would probably be reasonable to filter out link-local addresses (those starting with fe80) as well as the loopback interface.

The case of [fe80::1%lo0] is a little odd. Why does the loopback interface have a link-local address when there is ::1 allocated specifically for loopback? I guess that's just Apple's idea of Think Different. In any case I left listening on loopback in because it's not uncommon to connect to a client over loopback when doing experiments or verifying that a client is working.

[aldenml]

With all the details libtorrent takes care to avoid unnecessary work or waste, I don't feel confortable with the idea of opening non necessary sockets. If I'm correct, that means a full DHT node per UDP socket, small but not zero.

[ssiloti]

We already avoid creating DHT nodes on link-local addresses. One more thing we might be able to do is avoid creating nodes on "privacy" addresses. I'm not sure there's a cross-platform way of identifying them though.

[ssiloti]

Also IIRC macos is unusual in that it fully configures interfaces even when they are not in use (e.g. a wifi interface when there is also a wired connection). There may be a way of identifying the current "active" interface(s) and only listening on those. This would also require special care to make sure we get an IP change notification when the active interface changes.

[arvidn]

a, possibly, related consideration is if you have a VPN interface. chances are that you would like (and expect) to only listen on that interface. (I don't have a good answer to that)

[ssiloti]

I don't think we can handle VPNs in an automated fashion. It could be that the VPN is for privacy and the user wants that to be the only interface libtorrent uses, or it could be a VPN for work in which case they probably don't want libtorrent using the VPN at all.

[Seeker2]

qBittorrent v3.3.12 under Windows XP SP3 is listening on "extra" ports and multiple times on the same ports:

5/27/2017 4:45:56 AM - Options were saved successfully.
5/27/2017 4:45:55 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: UDP/65509
5/27/2017 4:45:55 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP/65509
5/27/2017 4:45:55 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: UDP/21933
5/27/2017 4:45:55 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP/21933
5/27/2017 4:45:55 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: UDP/65509
5/27/2017 4:45:55 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP_SSL/4433
5/27/2017 4:45:55 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP/65509
5/27/2017 4:45:55 AM - Embedded Tracker [OFF]
5/27/2017 4:45:55 AM - Encryption support [OFF]
5/27/2017 4:45:55 AM - Anonymous mode [OFF]
5/27/2017 4:45:55 AM - PeX support [OFF]
5/27/2017 4:45:55 AM - Local Peer Discovery support [OFF]
5/27/2017 4:45:55 AM - DHT support [OFF]
5/27/2017 4:45:55 AM - HTTP User-Agent is 'qBittorrent/3.3.12'
5/27/2017 4:45:55 AM - Peer ID: -qB33C0-
5/27/2017 4:45:55 AM - qBittorrent is trying to listen on any interface port: 65509
5/27/2017 4:45:55 AM - qBittorrent v3.3.12 started

In TCP View:

qbittorrent.exe TCP 0.0.0.0 4433    0.0.0.0 0   LISTENING
qbittorrent.exe TCP 127.0.0.1   21931   127.0.0.1   21932   ESTABLISHED
qbittorrent.exe TCP 127.0.0.1   21932   127.0.0.1   21931   ESTABLISHED
qbittorrent.exe TCP 0.0.0.0 65509   0.0.0.0 0   LISTENING
qbittorrent.exe UDP 0.0.0.0 65509   *   *

Next test with UPnP/NAT-PMP enabled, WEBUI enabled (port 8080), and embedded tracker enabled (port 9000):

5/27/2017 5:01:00 AM - UPnP/NAT-PMP: Port mapping successful, message: successfully mapped port using UPnP. external port: 65509
5/27/2017 5:00:58 AM - UPnP/NAT-PMP: Port mapping successful, message: successfully mapped port using UPnP. external port: 65509
5/27/2017 5:00:56 AM - The Web UI is listening on port 8080
5/27/2017 5:00:56 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: UDP/65509
5/27/2017 5:00:56 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP/65509
5/27/2017 5:00:56 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: UDP/22464
5/27/2017 5:00:56 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP/22464
5/27/2017 5:00:56 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: UDP/65509
5/27/2017 5:00:56 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP_SSL/4433
5/27/2017 5:00:56 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP/65509
5/27/2017 5:00:56 AM - qBittorrent is trying to listen on any interface port: 65509

In TCP View:

qbittorrent.exe TCP 0.0.0.0 8080    0.0.0.0 0   LISTENING
qbittorrent.exe TCP 0.0.0.0 65509   0.0.0.0 0   LISTENING
qbittorrent.exe TCP 0.0.0.0 9000    0.0.0.0 0   LISTENING
qbittorrent.exe TCP 10.0.0.18   22470   10.0.0.1    49152   ESTABLISHED
qbittorrent.exe TCP 127.0.0.1   22463   127.0.0.1   22462   ESTABLISHED
qbittorrent.exe TCP 127.0.0.1   22462   127.0.0.1   22463   ESTABLISHED
qbittorrent.exe UDP 0.0.0.0 1900    *   *
qbittorrent.exe UDP 0.0.0.0 22467   *   *
qbittorrent.exe UDP 10.0.0.18   22465   *   *
qbittorrent.exe UDP 0.0.0.0 65509   *   *
qbittorrent.exe UDP 127.0.0.1   22466   *   *
qbittorrent.exe UDP 10.0.0.18   1900    *   *
qbittorrent.exe UDP 127.0.0.1   1900    *   *

NEXT Test with UPnP, WEBUI, and embedded tracker disabled:

5/27/2017 5:08:37 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: UDP/65509
5/27/2017 5:08:37 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP/65509
5/27/2017 5:08:37 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: UDP/22648
5/27/2017 5:08:37 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP/22648
5/27/2017 5:08:37 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: UDP/65509
5/27/2017 5:08:37 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP_SSL/4433
5/27/2017 5:08:37 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP/65509
5/27/2017 5:08:37 AM - qBittorrent is trying to listen on any interface port: 65509

In TCP View:

qbittorrent.exe TCP 0.0.0.0 65509   0.0.0.0 0   LISTENING
qbittorrent.exe TCP 127.0.0.1   22647   127.0.0.1   22646   ESTABLISHED
qbittorrent.exe TCP 127.0.0.1   22646   127.0.0.1   22647   ESTABLISHED
qbittorrent.exe UDP 0.0.0.0 65509   *   *

Process Monitor shows the localhost (127.0.0.1) port actually has traffic on it (1-byte-at-a-time):

5:35:53.8034693 AM  qbittorrent.exe 3044    TCP Receive localhost:23460 -> localhost:23461  SUCCESS Length: 1
5:35:53.8034782 AM  qbittorrent.exe 3044    TCP Send    localhost:23461 -> localhost:23460  SUCCESS Length: 1
5:35:53.8059313 AM  qbittorrent.exe 3044    TCP Receive localhost:23460 -> localhost:23461  SUCCESS Length: 1
5:35:53.8059417 AM  qbittorrent.exe 3044    TCP Send    localhost:23461 -> localhost:23460  SUCCESS Length: 1
5:35:53.9071866 AM  qbittorrent.exe 3044    TCP Receive localhost:23460 -> localhost:23461  SUCCESS Length: 1
5:35:53.9071984 AM  qbittorrent.exe 3044    TCP Send    localhost:23461 -> localhost:23460  SUCCESS Length: 1
5:35:55.4069784 AM  qbittorrent.exe 3044    TCP Retransmit  localhost:23461 -> localhost:23460  SUCCESS Length: 0

Deluge v1.3.15 (which uses the same libtorrent 1.0.11.0) experienced the same problem:

deluge.exe  TCP 0.0.0.0 45107   0.0.0.0 0   LISTENING
deluge.exe  TCP 0.0.0.0 65505   0.0.0.0 0   LISTENING
deluge.exe  TCP 0.0.0.0 4433    0.0.0.0 0   LISTENING
deluge.exe  TCP 127.0.0.1   23157   127.0.0.1   23156   ESTABLISHED
deluge.exe  TCP 127.0.0.1   23156   127.0.0.1   23157   ESTABLISHED
deluge.exe  TCP 127.0.0.1   23164   127.0.0.1   23163   ESTABLISHED
deluge.exe  TCP 127.0.0.1   23163   127.0.0.1   23164   ESTABLISHED
deluge.exe  UDP 0.0.0.0 65505   *   *

Earlier example of same problem: https://qbforums.shiki.hu/index.php/topic,3982.msg24170.html#msg24170

[arvidn]

@Seeker2 I see duplicate listen_successful_alert notifications on "UDP/65509". However, your netstat output doesn't seem to indicate any problem. Could you please state what the issue is that you see and what you would expect instead (and why). Is it just the "successfully listening on.." duplicate?

It may make sense to post this as a new ticket as well, since the duplicate you refer to is different than what @aldenml was referring to in his report. To make it easier to understand your bug report, it may help to strip down the information you put in it to just highlight the problem, and make the libtorrent version prominent.

[aldenml]

Yes @Seeker2 this is another problem in itself, and depending on the version of libtorrent (like RC_1_1) and the way it is used, its probably that a "reopening of listening sockets" is taking place more than once.

[Seeker2]

@arvidn and @aldenml Yes, "reopening of listening sockets" appears to be taking place more than once/duplicates.

But also the established connections using "random" ephemeral ports happens...and they don't match what's in qBT's logs: 5/27/2017 4:45:55 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: UDP/21933 5/27/2017 4:45:55 AM - qBittorrent is successfully listening on interface 0.0.0.0 port: TCP/21933 These ALWAYS seem to appear between the "successfully listening on interface" messages for the proper listening port.

In TCP View: qbittorrent.exe TCP 0.0.0.0 4433 0.0.0.0 0 LISTENING qbittorrent.exe TCP 127.0.0.1 21931 127.0.0.1 21932 ESTABLISHED qbittorrent.exe TCP 127.0.0.1 21932 127.0.0.1 21931 ESTABLISHED

This occurred on both qBitTorrent v3.3.12 and Deluge v1.3.15 -- which both use libtorrent 1.0.11.0

[the8472]

We already avoid creating DHT nodes on link-local addresses. One more thing we might be able to do is avoid creating nodes on "privacy" addresses.

In my opinion the opposite should be done. If privacy extensions are enabled on that machine then the currently active one should be preferred over the permanent address. It blurs the accuracy on those "I know what you torrented last summer" sites a little at least.

My assumption is that privacy extensions are not enabled on server-class machines with stable addresses, e.g. in datacenters but are used on home PCs.

[Seeker2]

When I disconnected the interlinked 127.0.0.1 connections on consecutive ephemeral ports, qBitTorrent.exe's CPU usage shot up to using 1 full core (well virtual core, due to using hyperthreading)! This was qBT v3.3.13 using libtorrent 1.0.11.0 on an ancient Win XP SP3 box.

Deluge v1.3.15 using libtorrent 1.0.11.0 (on the same computer) does the same 100% cpu core thing, except it has twice as many interlinked 127.0.0.1 connections as qBT (4 instead of 2).

[arvidn]

When I disconnected the interlinked 127.0.0.1 connections on consecutive ephemeral ports

What do you mean by that? Do you mean you disabled the network device to cause the connections to close? what do you mean by "interlinked"?

[Seeker2]

I disconnect/"break" them by using TCP view (3rd party app) and selecting "Close Connection" -- it only disconnects the 2 connections on 127.0.0.1 that I had highlighted, not any of the others used by qBT on the same interface/network device. I am guessing it sends RST packets to that particular connection, since it's using TCP. (Much like how ComCast's Sandvine equipment worked on BitTorrent traffic between 2007-2010.)

I meant "interlinked" because the 2 ephemeral port ESTABLISHED connections are to each other. As shown in my previous posts above -- One's IN is the other's OUT.

Immediately after doing that, qBT instantly jumps to using a full virtual CPU core -- and even sometimes a tiny bit more...like ~26% cpu total (out of 4 virtual cores.) Deluge does this as well, which isn't surprising since both use libtorrent 1.0.11

uTorrent does not do this. Tixati does not do this. Even Transmission-Qt v2.84.9 does not do this, despite also creating persistent ESTABLISHED 127.0.0.1 connections on its start-up.

[arvidn]

when in this state of using a lot of CPU, does it still function? or does it appear to be hung/stuck somewhere? Do you still get logs? if so, would you mind increasing the log level/enabling as much logging as possible and report back what you see? I imagine that catching this in a debugger, it would be fairly simple to figure out what's going on, do you think you'll have a chance to do that?

[Seeker2]

It still functions...the 25% cpu usage then added to the "normal" cpu usage and it hit 50-56% cpu usage with 1 or 2 torrents actively downloading. My Intel i5 650 CPU clocked at 3.2 Ghz just doesn't seem fast anymore. :(

Changing outgoing connections to specific ports in qBT seemed to have no effect.

qBitTorrent's log reveals nothing changes when I close the 127.0.0.1 connections nor anything out of the ordinary after that. I already had qBT set to log normal, information, warning, and critical messages.

In Process Explorer for qbittorrent.exe under Handles, nothing changed when I closed qBT's 127.0.0.1 connections...and qBT rose to 25% CPU usage. The linked DLLs (all Microsoft ones!) didn't change either.

Recompiling the code with debugging on...I pretty much cannot do.

I wouldn't be surprised if the 127.0.0.1 connections are part of debugging testing in the first place. Or they are used as a workaround for the binding issues with VPNs/proxies -- if the "tripwires" break, the VPN or proxy is disconnected.

Deluge exhibits the same behavior as qBT (127.0.0.1 connections, 25% cpu jump when those are closed, continues working even after that) but seems to have even less logging information than qBT.

EDIT: When Windows goes into Sleep mode, it breaks all network connections and maybe even shut off network adapters. That might cause... High CPU usage after sleep https://github.com/qbittorrent/qBittorrent/issues/7104

[aldenml]

@ssiloti , per your comment

I left listening on loopback in because it's not uncommon to connect to a client over loopback when doing experiments or verifying that a client is working.

Do you think it is reasonable to avoid opening in link-locals that are not loopback?

[ssiloti]

No. It may not be common to use link-local addresses, but they still could conceivably be utilized and I don't see any good reason not to listen on them.

[aldenml]

I think the good reason to avoid them is as weak as the reason to have them by default, do you think that a configuration at least could be added (and false by default)?

[ssiloti]

What harm does it do to listen on them? They are legitimate addresses which may be utilized, so IMO there needs to be some material reason not to listen on them. I would consider the extra resources used by the listen sockets to be negligible.

[aldenml]

I read more about link-local IPv6 addresses and clarified some aspects of them. I now agree that these could in fact be used. Thanks for your comments.

@Seeker2 the issue you are talking about I think it's not related to this one