search

rakshasa / rtorrent

rTorrent BitTorrent client
https://github.com/rakshasa/rtorrent/wiki
GNU General Public License v2.0
4.07k stars 408 forks source link

DHT returns nonexistent peers #759

Open ghost opened 6 years ago

ghost commented 6 years ago

I have this trackerless magnet. When I add it in Transmission, after a couple of minutes the DHT search succeeds, the client connects with the other peers and begins to download. When I load the same magnet in rTorrent, the DHT search does return some IPs, but they are not the same IPs that I see in Transmission. More precisely, rTorrent always come up with weird IPs starting with 54.58; these are obviously not the wanted peers, nor are reachable addresses, therefore no actual peer is added and the magnet gets stuck. I tried stopping every torrent except the interested one and this is what I see in the logs (torrent hash & IPs are censored):

1531517725 I 53Fxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0ED->tracker_list: sending 'updated (group:4 url:dht://)
1531517785 I 53Fxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0ED->tracker_list: received 2 peers (url:dht://)
1531517785 I handshake_manager->54.58.AA.AA: Adding outcoming connection: encryption:13 message:3.
1531517785 I handshake_manager->54.58.BB.BB: Adding outcoming connection: encryption:13 message:3.
1531517845 I handshake_manager->54.58.AA.AA: Received error: message:7 network unreachable.
1531517845 I handshake_manager->54.58.BB.BB: Received error: message:7 network unreachable.

For the sake of completeness I should say that both rTorrent and Transmission resolve the same number of peers, e.g. Transmission finds three peers online => rTorrent finds three peers too, all of them wrong, all of them starting with 54.58.

I'm running rTorrent on a Raspberry Pi armv6l, and I've experienced the same exact behavior with: rTorrent 0.9.6 & libtorrent 0.13.6 from the Raspbian repos rTorrent 0.9.6 & libtorrent 0.13.6 built by me rTorrent 0.9.7 & libtorrent 0.13.7 from the latest commits, built by me, with and without the aligned flag (see issue #130)

Some lines from my config, even though I don't think they are relevant at all:

network.port_range.set = 50000-60000
network.port_random.set = yes
dht.mode.set = auto
protocol.pex.set = yes
trackers.use_udp.set = yes
protocol.encryption.set = allow_incoming,try_outgoing,enable_retry

Ironically, if I start transmission I can see that one of the peers is seeding this torrent with libtorrent 0.13.6, so I assume that this is a highly specific case.

pyroscope commented 6 years ago

These IPv4 are obviously hex-coded, so not actually 54.58.x.x/16.

ghost commented 6 years ago

Hex encoded? All the IPv4s I've seen in the logs are in the usual dotted decimal form... If I check the logs after adding a normal torrent with at least one tracker, I can see the handshake manager logging the same decimal addresses that show up in the peer section. The AA and BB you see up there were added by me in place of the actual digits because I didn't want to post an IP here.

ghost commented 6 years ago

Same exact behavior with rTorrent 0.9.6/libtorrent 0.13.6 on a totally different machine running Gentoo; I also tried with other trackerless torrents that I know for sure to be well-seeded. I really have no clue what is going on. Are you guys normally able to resolve trackerless torrents? Could this be related to the fact that I don't have (and I can't) set a port forwarding for DHT & bittorrent on my network?

chros73 commented 6 years ago

Are you guys normally able to resolve trackerless torrents?

Yes. Are you saying that downloading never starts? Can you post a magnet link that doesn't work for you?

... I don't have (and I can't) set a port forwarding ...

That indeed can be the problem (although I never used without it): wiki page entry

ghost commented 6 years ago

Yes. Are you saying that downloading never starts?

Exactly. Torrents is stuck since I can't connect to peers.

Can you post a magnet link that doesn't work for you?

magnet:?xt=urn:btih:53FC32C380D0F039011C7C410F52E1AE3C06B0ED

It's just a collection of Sci-Fi pics, so nothing controversial to post here. But my issue happens with every trackerless magnet or torrent file. I want to understand if port forwarding is strictly required with rTorrent (Transmission works fine even without, for example).

Edit: seeders for that torrent went offline magnet:?xt=urn:btih:e4be9e4db876e3e3179778b03e906297be5c8dbe This is a magnet for the latest Ubuntu - same issue: starts in Transmission, hangs in rTorrent, DHT discovers mysterious 54.58.x.x IPs.

mhertz commented 6 years ago

I also have issues at times with unpopular magnets without fall-back trackers added, and where deluge-console works. I just thought it was because rtorrent doesn't support uTP and i've read some clients favor uTP against TCP. Also, I know uTP features an extension to the protocol enabling NAT traversal, so can contact firewalled peers. Note, I don't have DHT port forwarded on deluge either(I'm on a new VPN provider not supporting port-forwarding - contrary to what most think - uploading can be done anyways, through outgoing connections as bittorrent spec is assymetrical).

I was thinking that maybe it was because other clients bootstrap there DHT-nodes at the beginning, as many times I go by a rtorrent-reinstall(which I set to wipe clean e.g. old DHT - I probably should change that in my script!), so empty DHT, or that it haden't been enabled for some time because I use auto like you for DHT. Maybe it would help if adding to .rtorrent.rc(and maybe also change DHT to ON? like most other clients):

schedule2 = dht_node_1, 5, 0, "dht.add_node=router.utorrent.com:6881" schedule2 = dht_node_2, 5, 0, "dht.add_node=dht.transmissionbt.com:6881" schedule2 = dht_node_3, 5, 0, "dht.add_node=router.bitcomet.com:6881" schedule2 = dht_node_4, 5, 0, "dht.add_node=dht.aelitis.com:6881"

(Thanks chros73 for the lines ;) )

It's probably unrelated based on your logs, so sorry for meddling :)

(Totally unrelated, I also have issues with magnets with fall-back trackers, because rtorrent imho is too unaggressive i.e. I believe that if pex is enabled and gets one hit, that don't even work, it stops connecting to the other trackers every 30 sec/3 times when under min_peers! I wish rtorrent had settings for changing tracker behaviour, as currently I need deluge-console installed(with enabled announcing to all trackers/tiers) for unpopular magnets, just to check that it's truly unpopular and not just a rtorrent-non-aggressive-issue.)

Edit: That link with ubuntu, didn't work with me, and said no DHT peers to search for - when enabling DHT to ON it receives DHT nodes and searches them, but doesn't start downloading, as found 0 peers from the searches. With deluge-console it starts up right away , full speed - 24 MB's second! Latest deluge and rtorrent-ps-ch.

Personally, i'm sad that I cannot rely on rtorrent solely, which i'd love! :( It's not about being more aggressive for better speeds per say, but just simply to be able to partake on torrents with only a small handfull of peers i.e. a compatibility issue. (Funny thing is that rtorrent by default is more aggressive than deluge in regards to failing trackers, where rtorrent reconnects in small intervals, and deluge just waits around for next re-announce long later - though i've changed that too in libtorrent-rasterbar settings with a deluge-plugin)

Hope i'm not coming forth as complaining or something here, as that's seriously not my intention! I'm very grateful for the work of you guys! (rakshasa, pyroscope and chros73) :)

grolongo commented 6 years ago

Wanted to post a new issue regarding UDP/DHT connections problems but a comment here will avoid redundant tickets as my issue seems related.

Since 0.9.7 (from Debian testing) I cannot use any magnet links as it fails to connect to any DHT servers.
I've followed carefully the tutorial and tweaked settings accordingly in the .rtorrent.rc but without success.

I've rolled back to 0.9.6 (from Debian stable), on the very same machine (WSL in my case) with my old config and magnet links worked straight away as expected.

If I can help with any logs I would be happy to provide them but as far as I looked they didn't show any relevant stuff to me unfortunately.

chros73 commented 6 years ago

@SovalouRT , sorry for the late reply. I just tested it with your magnet and a lubuntu iso magnet with the mentioned config snippet (using current rtorrent-ps-ch (based on v0.9.7)):

Conclusion, both are working here with the following condition (although yours needed about 30 minutes (!) to get the torrent file from a peer (similarly to utorrent v2.2.1)):

Note that I used this script to create fake torrent files from magnet links, although it shouldn't matter.

chros73 commented 6 years ago

@mhertz:

I'm on a new VPN provider not supporting port-forwarding

Try AirVPN next time :)

maybe also change DHT to ON

Back in the day (with v0.9.6 :) ) there were some issues (I forgot what they were) with this setting, and value auto work the best. But you can test it and report back after couple of months :)

I believe that if pex is enabled and gets one hit, that don't even work, it stops connecting to the other trackers every 30 sec/3 times when under min_peers!

There's a bug in rtorrent regarding to this, and the proposed fix is this (included in rtorrent-ps-ch). This is how it works with the fix:

receives DHT nodes and searches them, but doesn't start downloading

See the comment above.

@grolongo: did you open/forward ports?

grolongo commented 6 years ago

@chros73 no I didn't open/forward any ports

I could do that, but I don't understand why I would have to do it now for 0.9.7 to work when it was working great on 0.9.6 without touching anything on my router?

mhertz commented 6 years ago

Thanks chros73 for your input! :) I found a deal I couldn't resist with a lifetime subscription to a VPN for very cheap and which had good speeds, so I cancelled my old and accepted that there weren't any port-forwarding because of that, but of course it's not optimal I know, though only using public torrents(and no issues with deluge-console because of this).

I just don't get why getting one peer from PEX should stop connecting to other trackers(it's maybe a 0.5kb transfer or stalled), but that's the design the author intended and I will respect that of course :)

Anyway, i'm gonna test 0.9.6 now too, as it seems it works better for non-forwarded ports according to grolongo, but would just be sad to miss out on '-ps-ch' newer versions :) Thanks again!

Edit: Thanks alot for pointng me to the place in the source! :) I think that I maybe can change the PEX behaviour, but couldn't find the other stuff I wanted changed i.e. "max 3 times or if not getting 10 new peers" or "30 secs between", Not surprising as i'm no programmer and only do shell-scripting and haven't gone much beyond "hello world" when trying learning C :)

chros73 commented 6 years ago

@mhertz:

i'm gonna test 0.9.6 now too

Report back your findings :) (I have never used rtorrent with closed ports.)

@grolongo:

I could do that

Try out with it.

I don't understand why I would have to do it now for 0.9.7 to work when it was working great on 0.9.6

Well, me neither :) But there's a huge code change between these 2 releases (not just as the release notes suggests).

mhertz commented 6 years ago

@chros73

Report back your findings :) (I have never used rtorrent with closed ports.)

I just installed latest rtorrent-ps, which was easier than manually installing rtorrent-0.9.6 and libtorrent-0.13.6, but if that's not good enough(I know pyroscope adds alot), then I can test with v0.9.6 vanilla too. For me it's the same with the ubuntu link - initially it states no peers to search DHT with, as a clean install, and when restarting, then after 5 secs, it searches DHT(because of the lines from chros73) and e.g. states '9/24 nodes replied', even without any forwarding, but never starts downloading/getting actual peers(even though it states announcing 2/3 peers, after the DHT node search).

Edit: Okay, just in case I manually built rtorrent-0.9.6 and libtorrent-0.13.6, and the above done test is exactly the same.

chros73 commented 6 years ago

Thanks, that's what I thought, so it works the same as the new version. (And rtorrent-ps is fine to test with.) So how come that @grolongo has different result? :)

grolongo commented 6 years ago

Alright, I've tried a few things and nothing make it works.

I opened/forwarded the port set: network.port_range.set = 49164-49164 in my router:

ports ("les deux" = both tcp and udp, "équipement" = my desktop pc hostname)

I then completely turned off my PC firewall to make sure nothing got in the way.

Loading up three magnet links:

This is what happens (and I'm stuck like that for hours):

udpno

Now what's strange is using the older version 0.9.6 with my old basic config (that I used for years):

peer_exchange = yes
directory = ~/Downloads
session = ~/.rtorrent.session
check_hash = yes
port_range = 49164-49164
use_udp_trackers = yes
dht = auto
dht_port = 6881
encryption = require,allow_incoming,try_outgoing

all torrents load up straight away and download at full speed, plus without having to touch anything in my firewall nor having any opened/forwarded ports in my router.

If that may help, here is the new config I'm using for 0.9.7:

#############################################################################
# A minimal rTorrent configuration that provides the basic features
# you want to have in addition to the built-in defaults.
#
# See https://github.com/rakshasa/rtorrent/wiki/CONFIG-Template
# for an up-to-date version.
#############################################################################

## Instance layout (base paths)
method.insert = cfg.basedir,  private|const|string, (cat,"/home/grolongo/rtorrent/")
method.insert = cfg.download, private|const|string, (cat,(cfg.basedir),"download/")
method.insert = cfg.logs,     private|const|string, (cat,(cfg.basedir),"log/")
method.insert = cfg.logfile,  private|const|string, (cat,(cfg.logs),"rtorrent-",(system.time),".log")
method.insert = cfg.session,  private|const|string, (cat,(cfg.basedir),".session/")
method.insert = cfg.watch,    private|const|string, (cat,(cfg.basedir),"watch/")

## Create instance directories
execute.throw = sh, -c, (cat,\
    "mkdir -p \"",(cfg.download),"\" ",\
    "\"",(cfg.logs),"\" ",\
    "\"",(cfg.session),"\" ",\
    "\"",(cfg.watch),"/load\" ",\
    "\"",(cfg.watch),"/start\" ")

## Listening port for incoming peer traffic (fixed; you can also randomize it)
network.port_range.set = 49164-49164
network.port_random.set = no

## Tracker-less torrent and UDP tracker support
protocol.pex.set = 1
trackers.use_udp.set = 1
dht.mode.set = auto
dht.port.set = 6881

# Adding public DHT servers for easy bootstrapping
schedule2 = dht_node_1, 5, 0, "dht.add_node=router.utorrent.com:6881"
schedule2 = dht_node_2, 5, 0, "dht.add_node=dht.transmissionbt.com:6881"
schedule2 = dht_node_3, 5, 0, "dht.add_node=router.bitcomet.com:6881"
schedule2 = dht_node_4, 5, 0, "dht.add_node=dht.aelitis.com:6881"

## Peer settings
throttle.max_uploads.set = 100
throttle.max_uploads.global.set = 250

throttle.min_peers.normal.set = 20
throttle.max_peers.normal.set = 60
throttle.min_peers.seed.set = 30
throttle.max_peers.seed.set = 80
trackers.numwant.set = 80

#protocol.encryption.set = allow_incoming,try_outgoing,enable_retry
protocol.encryption.set = require,allow_incoming,try_outgoing

## Limits for file handle resources, this is optimized for
## an `ulimit` of 1024 (a common default). You MUST leave
## a ceiling of handles reserved for rTorrent's internal needs!
network.http.max_open.set = 50
network.max_open_files.set = 600
network.max_open_sockets.set = 300

## Memory resource usage (increase if you have a large number of items loaded,
## and/or the available resources to spend)
pieces.memory.max.set = 1800M
network.xmlrpc.size_limit.set = 4M

## Basic operational settings (no need to change these)
session.path.set = (cat, (cfg.session))
directory.default.set = (cat, (cfg.download))
log.execute = (cat, (cfg.logs), "execute.log")
#log.xmlrpc = (cat, (cfg.logs), "xmlrpc.log")
execute.nothrow = sh, -c, (cat, "echo >",\
    (session.path), "rtorrent.pid", " ",(system.pid))

## Other operational settings (check & adapt)
encoding.add = utf8
system.umask.set = 0022
system.cwd.set = (directory.default)
network.http.dns_cache_timeout.set = 25
schedule2 = monitor_diskspace, 15, 60, ((close_low_diskspace, 1000M))
pieces.hash.on_completion.set = yes
#view.sort_current = seeding, greater=d.ratio=
#keys.layout.set = qwerty
#network.http.capath.set = "/etc/ssl/certs"
#network.http.ssl_verify_peer.set = 0
#network.http.ssl_verify_host.set = 0

## Some additional values and commands
method.insert = system.startup_time, value|const, (system.time)
method.insert = d.data_path, simple,\
    "if=(d.is_multi_file),\
        (cat, (d.directory), /),\
        (cat, (d.directory), /, (d.name))"
method.insert = d.session_file, simple, "cat=(session.path), (d.hash), .torrent"

## Watch directories (add more as you like, but use unique schedule names)
## Add torrent
schedule2 = watch_load, 11, 10, ((load.verbose, (cat, (cfg.watch), "load/*.torrent")))
## Add & download straight away
schedule2 = watch_start, 10, 10, ((load.start_verbose, (cat, (cfg.watch), "start/*.torrent")))

## Run the rTorrent process as a daemon in the background
## (and control via XMLRPC sockets)
#system.daemon.set = true
#network.scgi.open_local = (cat,(session.path),rpc.socket)
#execute.nothrow = chmod,770,(cat,(session.path),rpc.socket)

## Logging:
##   Levels = critical error warn notice info debug
##   Groups = connection_* dht_* peer_* rpc_* storage_* thread_* tracker_* torrent_*
print = (cat, "Logging to ", (cfg.logfile))
log.open_file = "log", (cfg.logfile)
log.add_output = "info", "log"
#log.add_output = "tracker_debug", "log"

### END of rtorrent.rc ###

I'm really confused

chros73 commented 6 years ago

What OS, distrib do you use?

grolongo commented 6 years ago

@chros73 every cases example I described in previous posts on this thread happened on Debian on Windows Subsystem Linux, using rtorrent 0.9.6 from stable and 0.9.7 from testing repository.

Note I also have a Raspberry Pi with rtorrent 0.9.6 on it (can't use 0.9.7 as it hasn't been updated in the raspbian repo yet) and running with the basic config I posted just before.

The rtorrent on the Pi has also been running for years without any single problem using tracker and trackerless magnets, without ever opening/forwarding any port in my router and using these basic iptables settings:

*filter

:INPUT DROP
:FORWARD DROP
:OUTPUT ACCEPT

# Common Rules
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -s 127.0.0.0/8 ! -i lo -j DROP
-A INPUT -p icmp -m conntrack --ctstate NEW -j ACCEPT

COMMIT

if that can help.

mhertz commented 6 years ago

lol, at swedish site with a boat! :) The only thing i'll say is that when it for me says "no peers for dht search", then it's because I had an empty DHT, and even though the dht-nodes are updated after 5 secs, then when adding torrents after that, then it still won't work for some reason, so I restart rtorrent after adding the torrents and then it bootstraps and searches nodes(as the dht-nodes are updated with torrents loaded and not empty), but just didn't found peers for me in my try, for neither v0.9.7 or v0.9.6. The v0.9.6 that works for you, is that having another dht file than v0.9.7? If it has an old full DHT then that would be the difference i'd guess.

Btw, it's up to you if you open ports or not(and I don't because of my VPN provider), but as you get no incoming connections you will only be able to upload on outgoing connections, which I'd guess would somewhat limit your sharing and hence performance, as the bittorrent spec uses tit-for-tat algo which chokes you often if not sharing - popular torrents doesn't really matter as so many seeds available, in my experience(i've used a socks5 proxy for many years before using rtorrent/VPN, which also doesn't support incoming connections, without any issues either whatsoever and fast speeds).

Ohh, check your port is correctly opened also by e.g. canyouseeme.org

Unrelated, but there'a a bug I believe in v0.9.7, where it reports &IPv4 fields even for Ipv4 connections(semantics of an IPv6 tracker extension protocol), so there's a line you can set in your .rtorrent.rc to bind to correct IP and send out to trackers so you're connectable, and that is in the wiki by chros73. I run always rtorrent from scripts or shell-functions so I just used(changed to non-vpn IP):

rtorrent -i $(echo $(curl ifconfig.co))

Edit: Sorry i'm an idiot, forget last part, as unrelated without VPN, as proper IP is used i'd guess.

grolongo commented 6 years ago

@mhertz

The v0.9.6 that works, is that having another dht file than v0.9.7?

I think it does use a different one since I'm not using any of this in my old config for 0.9.6:

schedule2 = dht_node_1, 5, 0, "dht.add_node=router.utorrent.com:6881"
schedule2 = dht_node_2, 5, 0, "dht.add_node=dht.transmissionbt.com:6881"
schedule2 = dht_node_3, 5, 0, "dht.add_node=router.bitcomet.com:6881"
schedule2 = dht_node_4, 5, 0, "dht.add_node=dht.aelitis.com:6881"

but as you get no incoming connections you will only be able to upload on outgoing connections

INPUT/INCOMING connections have always been set to DROP/REJECT (except for some SSH / Seafile server ports from time to time) on both my router and local machines and I never had any problem downloading though.

@chros73 I'm going to try this: I'll do a clean install of my Debian WSL and download rtorrent 0.9.6 like I used to. Then I'll use my old config as normal and change it to the new one (still on 0.9.6 so) to see what happens.

This way I could tell:

I could have done the same using my old config on 0.9.7 but now you get error messages because of new syntax options (hence the new config).

I'll let you know.

grolongo commented 6 years ago

update: the new configuration works fine on 0.9.6, no port forwarding from my router needed nor specific firewall settings.

I just loaded a magnet link and it started to download straight away:

itworks

I still get [Unable to connect to UDP tracker.] as shown in the screenshot though.

So I guess that's a version issue, exact same .rtorrent.rc on the same machine works for 0.9.6 but not with 0.9.7, how can I troubleshoot that now?

chros73 commented 6 years ago

Are you sure that it was trackerless magnet? Is the above posted lubuntu iso working as well?

mhertz commented 6 years ago

So you now tested v0.9.6 with an empty or non-existent '/home/grolongo/rtorrent/.session/rtorrent.dht_cache', like with v0.9.7?

(Just making sure the difference isn't a big old DHT cache on v0.9.6 compared to v0.9.7)

Anyway, I guess the only thing to do then is compare the DHT logs between both runs/versions.

grolongo commented 6 years ago

@chros73

Are you sure that it was trackerless magnet? Is the above posted lubuntu iso working as well?

The torrent downloading in the last screenshot is coming from a magnet link on the pirate bay (AFAIK this website is not using tracker anymore).

So if it's not using tracker nor DHT am I downloading through peer exchange? Or am I confusing technologies here?

I loaded the lubuntu magnet and you're right it's not starting, can't even fetch the torrent metadata, just stuck at [No DHT nodes available or peer search.]

@mhertz

So you now tested v0.9.6 with an empty or non-existent '/home/grolongo/rtorrent/.session/rtorrent.dht_cache', like with v0.9.7?

Yes, I always remove all rtorrent related directories from my home folder before testing new version or new config to make sure it's clean.

Anyway, I guess the only thing to do then is compare the DHT logs between both runs/versions.

After what @chros73 said just above I think DHT just doesn't work on both version or configs, because of the lubuntu magnet I just tested which doesnt load: magnet:?xt=urn:btih:c1aa77dea674d71fbd85559034b6082b8434d36e&dn=lubuntu-16.04.3-desktop-amd64.iso

I must be confusing how it actually downloaded the torrent that worked (the one with the pixellated name in the screenshot), but if it downloaded without using tracker nor DHT, why wasn't isn't it downloading on 0.9.7 is a mystery.

mhertz commented 6 years ago

TPB uses fall-back trackers in there magnets, btw. You can check if peer-exchange got any hits by selecting the torrent, pressing arrow right and then check for: 'Peer exchange: enabled active(0/8)' e.g.

As said, when it says no peers to search DHT, as it does when newly added files on clean install, then restart rtorrent and it will search DHT because of the added nodes scheduled in .rtorrent.rc after 5 secs, but unfortunetly, it doesn't help with finding peers :( (I'm guessing the restart is needed because DHT isn't started on fresh start without torrents and so adding new nodes to a disabled DHT will not work I guess, or else I don't get why a restart, or having DHT to ON, is needed)

I also don't understand why it doesn't work in rtorrent, but works fine in e.g. deluge-console, but there must be some subtle differences in the DHT implementation.

Anyway, sorry I cannot help you out with the issue, and i'm also personally sorry that I(/you) found another use-case which makes rtorrent a not good candidate for me unfortunately - I wish I knew C++ better than my "super elite skills" of being able to make a CLI calculator-type app, lol :)

I tried adding in two extra DHT nodes additionally, as that is what deluge and qbittorrent uses(one of chros73 ones weren't used by deluge/qbittorrent, but I kept that too), but still it doesn't work, but used little higher numbers after the restart i.e. 13/32 peers replied it stated this time, but still no go :( It's also strange that deluge/qbittorrent shows much greater number of nodes/peers at startup from bootstrapping DHT, but rtorrent even with these two extra nodes added, never shows more than low 30's, and which was my thinking of adding in the extra nodes, although I don't know if the two set of numbers are even remotely related. Again deluge-console with cleaned-out DHT, so starting from scratch like rtorrent, starts downloading within 2 seconds with 30 connected seeds and 1 peer and full speed(no forwarding going on, neither regular nor DHT ports).

A workaround is, I'd guess, to use danfolkes magnet2torrent.py script and set that as handler for magnets. I know you can use a simple shell-script to do the same, but that just makes a meta-torrent file which rtorrent supports, but no other client because it's not a true torrent, but just a torrent with the infohash from the magnet. The magnet2torrent.py script uses libtorrent-rasterbar through it's python-bindings(so need python-libtorrent or python3-libtorrent) to make a true torrent and retrieves metadata from the actual swarm so I guess it should work. Btw, I added a few lines to danfolkes version so that it after making the torrent will open the torrent with your standard defined torrent app and then delete the torrent-meta file afterwards: https://pastebin.com/dl/pujkSz56 (Place it somewhere under PATH, make it executable, and add as handler for magnets, but not torrents which should be kept as before set to rtorrent). Or use the regular version to run manually: https://github.com/danfolkes/Magnet2Torrent/blob/master/Magnet_To_Torrent2.py (There's another fork of that by LordAro which also can be set to be opened by torrent-client afterwards, but that needs extra commands to do that, in addition to the magnet-url, so it would need a shell-script for that if used as magnet handler and that's why I didn't used that, but the original version instead. If running it manually, then that would probably be the preferred version to use: https://github.com/LordAro/Magnet2Torrent/blob/master/Magnet2Torrent.py )

Sorry for long-winded'ness :)

Edit: Sorry, the above workaround didn't work after I just tested it, sorry about that, it just keeps being stalled at "downloading metadata - this can take a while" - normally it's only a few secs it takes. I made the changed version recently and posted on deluge-forum because of an issue with deluge not supporting magnets only torrents when using labelplus plugin for auto-labeling/sorting(using reg-ex rules for file-extensions etc).

Edit2: Still doesn't fix the rtorrent issue, but here's links for fixed magnet2torrent.py which now works with magnets without fallback-trackers also. The manual version I added a link to, means to not auto-load into default torrent-client afterwards, for use as manual tool:

https://gist.github.com/mhertz/01842039bd9d056001c82d16e83dc4e4

Manual:

https://gist.github.com/mhertz/c66dcb714c44d4725bd22f18003ae554

chros73 commented 6 years ago

I think DHT just doesn't work on both version or configs

That's correct :)

Conclusion is above.

I've also updated the corresponding wiki page section.

grolongo commented 6 years ago

@mhertz

TPB uses fall-back trackers in there magnets, btw. You can check if peer-exchange got any hits by selecting the torrent, pressing arrow right and then check for: 'Peer exchange: enabled active(0/8)' e.g.

So that would explain I was able to download my file. I'll check the pex as you suggested, thanks! EDIT: indeed it fluctuates between 6/8 8/8

13/32 peers replied it stated this time, but still no go

I remember having this too on 0.9.7 and still no download just like you :/

@chros73

So if for some reason I never noticed I was not using DHT (because I never opened any ports for rtorrent), I guess I was using fallback mechanisms to download from TPB.

But now, how come I'm able to download fine from TPB using 0.9.6 (like in the screenshot with pixelated torrent title) but not in 0.9.7 using the same .rtorrent.rc? Does that make sense?

chros73 commented 6 years ago

But now, how come I'm able to download fine from TPB using 0.9.6 (like in the screenshot with pixelated torrent title) but not in 0.9.7 using the same .rtorrent.rc? Does that make sense?

Honestly I don't know, but I/we could download those magnets that include trackers using 0.9.7 without opening any port. The only thing I can think of is your protocol.encryption setting, you can try out with this (note that it allows unencrypted connections): protocol.encryption.set = allow_incoming,prefer_plaintext,enable_retry

grolongo commented 6 years ago

The only thing I can think of is your protocol.encryption setting, you can try out with this (note that it allows unencrypted connections): protocol.encryption.set = allow_incoming,prefer_plaintext,enable_retry

Well, I just retested and even with no encryption, download never starts on 0.9.7 This is what I have (same torrent from TPB that worked on 0.9.6 with an identical .rtorrent.rc) and it's stuck like this since an hour atm:

notworking

I guess I'm stuck on 0.9.6 for now

mhertz commented 6 years ago

Yeah, grolongo sorry again that I cannot help either! :( Well, there's also magnet/dht issues with v0.9.6, as chros73 and I reported previously in this thread, but if it works better for you, then of course that's the only important thing :)

It was also reported little over a year ago an issue with magnets not working as good as deluge or transmission, in a report by mpv developer haasn :raised_hands: That report where with v0.9.6 also.

https://github.com/rakshasa/rtorrent/issues/602

Btw, I fixed the magnet2torrent.py script I mentioned earlier(added DHT-nodes for bootstrap, restarted DHT and added 2 sec break before feeding URI to libtorrent), but the resulting true torrent-file didn't start download in rtorrent either, as I should already have known! I don't know why I thought it would magically work while still without trackers, sorry! (I thought maybe some DHT-peers would get recorded in the torrent - I haven't read up on that part honestly)

Anyway, if you need magnet-links tested for v0.9.6 or v0.9.7 from swedish boat-sites or alike :) or something else I could help test, then do drop me a line! :)

Good luck!

Btw, the DHT-crawling search sites are becoming exceedingly popular, as they have many more results to show, which is great for unpopular and/or old/rare content, and most of these sites don't use fallback-trackers in there magnets! However, as they are just the same 4 or 5 standard used trackers, then there's no guaranty either that it has the stuff anyway, especially as often used for rare content, so maybe not so much help anyway, as the DHT then still would be need to be functioning properly regardless.

Also, I was thinking that it would be trivial to e.g. extend the standard script used for making fake torrents from magnets, to add 4-5 fall-back trackers to them, but still it wouldn't work always, only sometimes, which doesn't make it a full-proof solution unfortunately.

I just tested aria2c, and that works too with the lubuntu magnet above and no forwarded ports.

mhertz commented 6 years ago

Sorry for off-topic and just a very quick question and I hope none of the involved parties mind :)

@chros73, do you know what it means when defining a choke-groups's 'strings.tracker_mode' setting to 'aggressive'(instead of 'normal'), as i've also seen you've done in your configs? I'm referring to that normally we only have the tracker aggressiveness behavior very slightly togglable by raising 'throttle.min_peers.normal.set', but was wondering what that 'aggressive' option further was about? Thanks in advance.

grolongo commented 6 years ago

I just tested aria2c, and that works too with the lubuntu magnet above and no forwarded ports.

I think I will fallback to this aswell. With my 1 Gbps fiber internet I don't need to let rtorrent open on a box overnight for a download to finish anymore. Popular torrents end up downloading in less than 10 minutes even for files over 5 Go now.

Still, for a different case, rtorrent is much better for seeding and leeching on the long term with many torrents loaded under a tmux/screen session.

I see aria2c as the wget/curl of torrent/magnet file for single downloads, and that is fine for me at the moment. I'll try future rtorrent versions when they come out to see if it works again for me!

Anyway thanks a lot for the help @chros73 & @mhertz

mhertz commented 6 years ago

You're most welcome, and sorry nothing came out of it :( (From my part i mean, - chros73 identified what needed to be done to fix it at least).

Indeed nothing compares to rtorrent in tmux, it's lighweightness combined still with it's abundance of internal commands for scripting whatever your harts content(not even talking about the mind-blowing awesomeness that is the pyrocore tools combined with rtorrent-ps[-ch])! If only rtorrent opened up for tuning it alittle more in terms of aggressiveness, like libtorrent-rasterbar, then it would be a wet-dream of mine lol :) Anyway, aria2c is pretty nice and can be run either in tmux or as daemon and then you add more torrents or control stuff from xmlrpc calls(like rtorrent). There's good documentation e.g. for controlling aria2c from python(through xmlrpclib). I made a small script for adding torrents/magnets to running daemon and another for querying torrents from daemon. Keep in mind that the lubuntu magnet worked, but took 10 secs to begin I believe or thereabout.

chros73 commented 6 years ago

do you know what it means when defining a choke-groups's 'strings.tracker_mode' setting to 'aggressive'(instead of 'normal')

I don't remember :) But quickly searching through libtorrent project, I haven't seen any place where this setting is used :) All the rest of the choke group settings are working as they should, I assume you went through the explanation, this is one of the best feature of rtorrent :)

mhertz commented 6 years ago

Indeed, that's my source ;) Amazing work btw! Thanks again for all your help and input mate, I really appreciate that! :)

jtl999 commented 6 years ago

I can confirm with rTorrent 0.9.6 (using PS mod) and rTorrent 0.9.7 (vanilla) running on Debian that DHT doesn't seem to work well. I configured DHT, ensuring my incoming port and DHT port (probably unneeded) had incoming connections working properly and tested with my own torrents without any trackers seeded from my home connection to my "seedbox" host running rTorrent with a different public IP, and nothing. All torrents get stuck fetching the "metadata" and it doesn't acquire any other information.

I am willing to test and provide more information if needed.

wberrier commented 5 years ago

Also seeing this on el7 with rtorrent 0.9.6. Funny thing is, some magnet links seem to randomly work, others don't.

The ones that don't will attempt to "resolve" for hours before I finally give up. I try the same magnet link (on the same machine) with transmission, and it will usually start downloading in < 10 minutes.

chros73 commented 5 years ago

Just use aria2c to get the torrent file into your watch dir, like this script does.

ghost commented 5 years ago

I had the same difficulties as everyone above - until I correctly opened/forwarded my ports within the "port_range =" value in the rtorrent config. Once this range was opened on my router the DHT instantly worked. FWIW. This is on 0.9.7