UDP trackers and SOCKS5 support

[alex-druzhinin]

libtorrent can't connect to UDP trackers if SOCKS5 proxy used.

It calls tcp/accept and tcp/bind on the proxy. But when it try to send UDP packet to tracker - error operation_not_supported returned.

I can see the issue in master branch - in session_impl.cpp, method send_udp_packet_hostname. It want to use a socket from m_listen_sockets, but there are no sockets for UDP tracker. It should open a new UDP socket and use UDP ASSOCIATE, but instead returns error.

RC_1_1 also can't connect to UDP trackers, but looks like it's implemented in other way.

Is master branch stable enough to use it for a project? Or better we should stay on RC_1_1?

[arvidn]

It calls tcp/accept and tcp/bind on the proxy.

What do you mean by that? those aren't SOCKS5 commands.

What is the symptom you see? Did you capture packets to see what SOCKS5 commands are sent?

Do you get any error alerts listening to a UDP socket?

[alex-druzhinin]

Yes, tcp/accept and tcp/bind are from SOCKS server's log. I'm using Dante 1.4.1 without authentication.

Libtorrent at start connects to the server and send BIND command. Server responds "Succeeded".

When a magnet link added, it try to connect to trackers. Errors in log:

Log: TORRENT_LOG - torrent_log_alert - : ==> TRACKER REQUEST "udp://tracker.leechers-paradise.org:6969" event: started abort: 0

Log: LOG - log_alert - send_udp_packet_hostname

Log: TORRENT_LOG - torrent_log_alert - : ==> UDP_TRACKER_CONNECT [ failed: Operation not supported on socket ]

Log: TORRENT_LOG - torrent_log_alert - : *** tracker: "udp://tracker.leechers-paradise.org:6969" [ tiers: 0 trackers: 0 found: 0 i->tier: 0 tier: 2147483647 working: 1 fails: 0 limit: 0 upd: 1 ]

Log: TORRENT_LOG - torrent_log_alert - : *** update tracker timer: next_announce < now 0 m_waiting_tracker: 0 next_announce_in: -633435269

And no packets sent to proxy at this time.

Please find full log and Wireshark capture here - https://drive.google.com/open?id=0Bzflclj81dQkVk92a2Rmc0ZGbE0

Only few packets are sent to proxy at startup, then no network activity.

Torrents with HTTP trackers works fine on the same proxy.

[arvidn]

hm.. i need to do some digging. this looks suspicious though:

Log: LOG - log_alert - UDP error: 0.0.0.0:0 (89) Operation canceled

[alex-druzhinin]

Looks like there are two changes needed to get it work:

• set proxy for UDP socket in session_impl::setup_listener(): ret.udp_sock->set_proxy_settings(proxy());
• accept listen_socket_t with only UDP socket in check after setup_listener call in session_impl::reopen_listen_sockets(): if (!ec && (s.sock || s.udp_sock))...

Also in udp_socket::bind() I would use m_bind_port = m_socket.local_endpoint().port(); to get actual binded port.

[arvidn]

thanks for looking into this. When I get a chance I will prepare a PR for this along with a test (unless you beat me to it :) )

[arvidn]

would you mind taking a look at this patch? https://github.com/arvidn/libtorrent/pull/1431

[alex-druzhinin]

thanks for the update. I can't test it directly - I'm using libtorrent through Jlibtorrent, which is linked to master. But I can't see set_force_proxy call near set_proxy_settings for the UDP socket. Would it be set correctly when session initially have force_proxy?

[arvidn]

before I added support for udp-associate in the socks proxy in libsimulator setting force_proxy made it fail, so I feel pretty confident that setting does get set correctly.

[aldenml]

@alex-druzhinin ping me when this gets merge (and RC_1_1 into master) to release a jlibtorrent for you.

[arvidn]

@aldenml pushed and merged

[aldenml]

@alex-druzhinin 1.2.0.6-beta1 is out (including https://github.com/arvidn/libtorrent/pull/1436) if you want to give it a try

[alex-druzhinin]

@arvidn @aldenml thanks for the update!

1.2.0.6-beta1 works perfect, at least on Mac.

[alex-druzhinin]

Amm, but now it doesn't work. Not sure what is going on - I did download one torrent using magnet link (only UDP trackers). Now it's seeding and can collect seeds. But it's not send any UDP packets.

Other magnet links can't be downloaded, also no UDP traffic.

P.S. ret.udp_sock->set_proxy_settings(proxy()); after ret.udp_sock->set_force_proxy in session_impl.cpp fixes the issue.

[aldenml]

can you reopen de issue?

[arvidn]

@alex-druzhinin Do you know why set call to set_force_proxy() solves the problem? what I'm worried about is that you don't have the force_proxy setting enabled, and calling this function disables it just in time to allow libtorrent to not use the proxy (and hence obscuring the fact that the proxy still doesn't actually work)

[arvidn]

does this fix it? https://github.com/arvidn/libtorrent/pull/1482

[arvidn]

@aldenml @alex-druzhinin would you be able to confirm this patch? I feel pretty confident with it in case you'd want to build out of RC_1_1

[aldenml]

Hi @alex-druzhinin, jlibtorrent 1.2.0.6-beta2 is out, including https://github.com/arvidn/libtorrent/pull/1482 on top of master (see https://github.com/aldenml/libtorrent/tree/master-force-proxy), let us know

[aldenml]

@arvidn would you consider the merge even without confirmation?

[alex-druzhinin]

Sorry for long silence.

jlibtorrent 1.2.0.6-beta2 and beta3 does not work. It writes "giving up on binding listen sockets" to log and can't send UDP packets - m_listen_sockets is empty.

I can't find the master-force-proxy branch, working with arvidn's master.

I'm always use force_proxy and anonymous_mode with socks proxy. Force_proxy applied correctly after socket open, while proxy settings doesn't. So I add proxy settings after force_proxy. And add sockets with only udp_sock to m_listen_sockets.

In current master with such m_listen_sockets it also crashes in update_peer_tos().

Feel free to check my version of session_impl in my fork - https://github.com/alex-druzhinin/libtorrent/blob/master/src/session_impl.cpp - it works for my case.

[aldenml]

@alex-druzhinin I combined your branch with this PR here: https://github.com/arvidn/libtorrent/pull/1518, I will let you know when beta4 with this branch is released.

@arvidn I know it would be nice to have this fixed in RC_1_1, but I rather back port from master since it's faster to test

[arvidn]

sure. how can I see the patch that fixes it?

[arvidn]

oh, I see.. that's your PR :P

[arvidn]

@alex-druzhinin your patch is in master now. does it mean it works? I take it you didn't investigated why it isn't working in RC_1_1, and it's sufficiently different from master for that to be simple. is that right?

[aldenml]

Hi @arvidn I assume he is waiting for another jlibtorrent beta release, I have pending a proper port of https://github.com/arvidn/libtorrent/pull/1482 to master (in a way that does not trigger undefined behavior)

[arvidn]

was the issue self-assignment of m_proxy_settings?

[aldenml]

more or less, in the sense that at the time set_force_proxy is called, there were never called set_proxy_settings (and it will be not called later). The configuration is never set correct.

[aldenml]

@alex-druzhinin I believe https://github.com/arvidn/libtorrent/pull/1563 completes your fork changes. @arvidn now I'm not sure that https://github.com/arvidn/libtorrent/pull/1482 is correct for master.

[alex-druzhinin]

@aldenml yes, #1563 looks good, this is the needed part @arvidn You are right - I didn't find solution for RC_1_1 because I can't debug and test it now.

[aldenml]

Hi @alex-druzhinin, jlibtorrent-1.2.0.6-beta4 is out, do you mind test it and confirm if it fix the issue?

[alex-druzhinin]

@aldenml jlibtorrent-1.2.0.6-beta4 is working. At least on Mac and Win32. Thank you, this is great!

[scakemyer]

Was this fixed or not in 1.1.2?

[arvidn]

@scakemyer I don't believe so. the fix in master did not apply to RC_1_1, and I don't believe there was anyone testing RC_1_1. Do you experience this issue? if so, would you be willing to test patches?

[scakemyer]

I don't use the feature personally, but it seems we still do in scakemyer/plugin.video.quasar#635 and we'll need to get that resolved before v1.0. There's a few users I can send test builds to that should be able to test patches.

[scakemyer]

I'm the first one to get annoyed by someone asking for updates but... any updates?

[h-2]

I have also seen this since ever and it would be great to see it finally fixed. Thanks for your work on the library!

PS: or was this supposed to have been fixed with 1.1.2? Because it is still not working with 1.1.3...

[arvidn]

It does work in 1.1.3. It was fixed in december 2016. Here's the test that passes: https://github.com/arvidn/libtorrent/blob/RC_1_1/simulation/test_socks5.cpp#L170

Do you perhaps use some configuration that's different than what's exercised by the test? If the error you get is "not supported", are you confident your SOCKS5 server supports UDP? (some don't)

[h-2]

@arvidn Thanks for the quick reply. I am using libtorrent-rasterbar through qbittorrent. IIRC UDP tracker communication has never worked through the SOCKS5, however the proxy provider claims that UDP works and I know it's a dante server (which is supposed to support UDP). Anyway I can check easily with standard unix tools whether UDP is supported? nmap -sU -p 1080 IPADRESS reports the UDP port as closed, but I know that nmap is unreliable for UDP...

[arvidn]

I think the most reliable way to understand what's going on is to wireshark the TCP and UDP traffic to the SOCKS proxy.

[h-2]

@arvidn Thanks for your reply. Before I get all invested in a long debugging session (which I could do eventually, but not right now unfortunately): Is there some small tool that I can use client-side to check if UDP-through-SOCKS works? E.g. can I configure fetch, ftp, wget, curl or any other standard unix tool to establish a connection via UDP via SOCKS? Or is there some small C snippet that would do this? Because then I could check immediately whether this is a libtorrent issue or one on the server (in which case I would just delegate this to the service provider).

[shvchk]

@alex-druzhinin could you please provide Dante config you've been testing this on? I can't connect to UDP trackers with qBittorrent 4.0.1, libtorrent-rasterbar 1.1.5 and Dante 1.4.1. Here is my /etc/danted.conf:

logoutput: /var/log/sockd.log

internal: x.x.x.x port = 1080
external: y.y.y.y

socksmethod: pam none
clientmethod: none

user.privileged: proxy
user.unprivileged: nobody
user.libwrap: nobody

client pass {
        from: x.x.x.0/24 to: 0.0.0.0/0
        method: none
}

client block {
        from: 0.0.0.0/0 to: 0.0.0.0/0
        log: connect error
}

socks block {
        from: 0.0.0.0/0 to: lo0
        log: connect error
}

socks pass {
        from: x.x.x.0/24 to: 0.0.0.0/0
        protocol: tcp udp
}

socks block {
        from: 0.0.0.0/0 to: 0.0.0.0/0
        log: connect error
}

[alex-druzhinin]

@shvchk, here is our sockd.conf:

internal: 127.0.0.1 port = 1080
internal: eth0 port = 1080
external: eth0
method: username
user.privileged    : root
user.notprivileged : root
compatibility: sameport
extension: bind
client pass {
        from: 0.0.0.0/0 port 1024-65535 to: 0.0.0.0/0
}
pass {
       from: 0.0.0.0/0 to: 0.0.0.0/0
       command: bind connect udpassociate bindreply udpreply
       protocol: tcp udp
       proxyprotocol: socks_v5
}
pass {
        from: 127.0.0.1/8 to: 0.0.0.0/0
        command: bind
        protocol: tcp udp
        proxyprotocol: socks_v5
}
pass {
        from: 0.0.0.0/0 to: 127.0.0.1/8
        command: bindreply udpreply
        protocol: tcp udp
        proxyprotocol: socks_v5
}

[ichorid]

I have a report on this problem. Using libtorrent 1.1.6 client_test app. My test setup: Seeder (client_test) + opentracker tracker <-> SOCKS proxy (ssh -N -D 0.0.0.0:1080 test@localhost) <-> Leecher (client_test).

Test torrent file includes a test tracker URL: http://test1.local:6969/announce When downloading directly, both UDP and TCP - based tracker protocols work. These are checked by shutting down respective ports/protocols with iptables. Which means, that by default test_client tries to connect to the tracker by TCP, and if that is not possible, it would try UDP tracker protocol. However, when connecting through SOCKS proxy, if TCP is blocked at the proxy exit, it would not try UDP tracker connection. If URL is formed as udp://test1.local:6969, it would still try to connect to tracker through proxy using TCP tracker protocol.

This means that when libtorrent uses SOCKS proxy, it uses TCP tracker by default, even if URL is UDP-based. And if TCP connection fails, it will not try UDP tracker connection.

EDIT: Apparently, SSH would not forward UDP packets. Redid the same tests with RC_1_1 and 3proxy socks proxy, checking the usage of uTP with client_test -y flag. The result is the same: uDP packets are going OK through the proxy, but connecting to the tracker through UDP protocol via proxy is impossible.

client_test log:

[Apr 06 15:26:55] test.file (udp://test1.local:6969) sending announce (started)
[Apr 06 15:27:06] test.file (udp://test1.local:6969) (-1) timed out "" (1)

One interesting detail is the wireshark dump on router's interface has one loose UDP packet going from leecher to router (SOCKS proxy) unproxied: 10.0.3.5:6881 -> 10.0.3.4:33236 It's payload is 34 bytes:

0000   00 00 00 03 0b 74 65 73 74 31 2e 6c 6f 63 61 6c  .....test1.local
0010   1b 39 00 00 04 17 27 10 19 80 00 00 00 00 f1 c9  .9....'.........
0020   f6 b8                                            ..

test1.local is the DNS name of the tracker we're trying to connect to. So, it has test1.local and then 2 bytes 1b 39 that is 6969 - exactly the port number of the tracker. It seems like libtorrent is incorrectly sending some stuff to SOCKS proxy in form of a UDP packet.

EDIT2: Libtorrent (at the moment) does not use fallback from UDP to TCP and vice-versa for trackers. In my tests I was misled by libtorrent caching seeders in its resume files. And 3proxy does not serve UDP-hostname lookups. Use danted instead.

[arvidn]

Yeah, I was suspecting this was related to proxying the hostname lookup. I think you should be able to confirm this by disabling the proxy_hostname setting. Then the IP should be resolved locally and the SOCKS5 packet use the IP instead of hostname.

Unfortunately there's no simulation for UDP trackers, so it won't be easy to write a test for this. But I suspect the fix is simple

[arvidn]

As far as I can tell, that SOCKS5 UDP header is correct. Looking at RFC 1928, here's how it breaks down:

0000   00 00 00 03 0b 74 65 73 74 31 2e 6c 6f 63 61 6c  .....test1.local
0010   1b 39 00 00 04 17 27 10 19 80 00 00 00 00 f1 c9  .9....'.........
0020   f6 b8

00 00 - reserved, must be zero 00 - fragment number. 0, as this is the first and only fragment 03 - address type for destination. 3 means domain name, as opposed to IPv4 or IPv6 address 0b - length prefix of domain name string, 11 characters test1.local - domain name 1b 39 - destination port

followed by the payload, which at appears to be a valid UDP tracker packet, as it starts with 0x41727101980.

Are you sure 3proxy supports proxying hostnames?

[arvidn]

also, I don't think libtorrent is supposed to automatically try an http tracker as UDP in case it fails. I can't find any code that does that at least. Are you sure that's the behaviour you see?

[Youngman22]

Its well

[ichorid]

@arvidn, you were right, it was 3proxy's fault. I'm sorry on raising false alarm on that one. I retested everything with dante proxy, and UDP-hostname resolution worked OK. (BTW, danted UDP performance sucks). However, wouldn't you consider adding a feature of falling back to UDP if HTTP tracker connection fails, and vice-versa? Would that go well with libtorrent's philosophy, or should it be implemented at the level of the client itself?

[h-2]

I am little confused by the last posts, which of the following is true / do you assume:

1. libtorrent has always worked with UDP trackers over SOCKS5 if the proxy server is configured properly
2. there was an issue in libtorrent, but now it's fixed
3. libtorrent only works with UDP trackers over SOCKS5 if the name resolution happens locally

If 1.: what do I need to tell my service provider? I know they are using dante, it is set up to do UDP, I don't have issues with other applications that are proxied and other types of UDP connections of libtorrent (peer traffic) seems to work through the proxy.

If 2. : Which version?

If 3.: For me communication with UDP trackers also doesn't work if I give the tracker's IP instead of hostname so I pretty sure this is something else...

Thank you very much!

PS:

% pkg info | grep torrent
libtorrent-rasterbar-1.1.6_2   C++ library implementing a BitTorrent client
qbittorrent-3.3.16_2           Bittorrent client using Qt4/5 and libtorrent-rasterbar

[arvidn]

libtorrent has always worked with UDP trackers over SOCKS5 if the proxy server is configured properly

I believe this is more or less true. There has been issues in socks5 support in the past, but I don't believe there are any unresolved issues right now.

what do I need to tell my service provider?

It depends on what problem you're having. You have to wireshark your interaction with the proxy and compare the messages with the specification. For instance, in the case above it appears 3proxy did not support the domainname address type (3), but required ipv4 (1) or ipv6 (4) types.

If all the SOCKS5 messages sent from you to the proxy are correct according to the standard, but you get incorrect responses or if relaying doesn't work, that's what you tell your service provider.

If you see invalid messages being sent to the proxy, please file a bug report with libtorrent.

there was an issue in libtorrent, but now it's fixed

libtorrent only works with UDP trackers over SOCKS5 if the name resolution happens locally

Which version?

I don't remember off-hand. Have a look at the Changelog or the revision history of src/udp_socket.cpp (if you're primarily interested in UDP support).

libtorrent only works with UDP trackers over SOCKS5 if the name resolution happens locally

No, this is not a true general statement. It appears to be the case with 3proxy, from what I gather above 3proxy doesn't support the domainname address type.

For me communication with UDP trackers also doesn't work if I give the tracker's IP instead of hostname so I pretty sure this is something else...

Do you end up using the domainname address type (3) where the hostname is an IP address, or are you actually using the IPv4 address type (1)?

Unless it's the latter, it doesn't really provide a useful sample.