Private magnet links

[AllSeeingEyeTolledEweSew]

I'd like to propose adding a &private=1 parameter to magnet links, to be friendlier to private trackers.

In general private trackers distrust magnet links, though the reasons aren't always clear. It is possible to use them though, and I'm a member of at least one private tracker which does.

The normal usage seems to be that the magnet links will always contain a &tr= parameter (the tracker URL will have embedded "passkey"-style auth info). A client will connect to the tracker for a list of peers, retrieves the metadata from the peers using the ut_metadata extension, and downloads the torrent normally.

However I notice that my client (deluge 1.3.15 + libtorrent 1.1.4) will still do DHT searches for peers before the metadata is downloaded. This makes sense, although all private trackers enforce that the private flag is set in the infodict, the client won't see this until peers are connected and the infodict is transferred, so it couldn't know the torrent is private.

This leaks information outside the domain of the tracker, which is quite bad. At the least, it's a situation ripe for undefined behavior (would the client disconnect DHT-sourced peers on discovering the torrent is private?). In practice I do see DHT peers are sometimes discovered and connected before the metadata is downloaded, as some private torrents do get reposted publicly. In the theoretical nightmare case, DHT crawlers could use this to leech data, or expose tracker members. In my experience theoretical nightmares are the favorite pastime of tracker administrators.

So I propose the &private=1 parameter to magnet links having the same semantics as BEP 27, i.e. the client may only connect peers returned from the named tracker for metadata transfer. It's an error for &private=1 to be present without &tr=.

I'd also lke to define the behavior that when a torrent is initialized as public without metadata (i.e. magnet link with no &private=1) but is later discovered via metadata to be private, the client should mark the torrent with an error, and ... attempt to undo the damage somehow. I'd like to say the client should disconnect from all peers that are not sourced from the tracker, but I'm not sure this does any good, since new peers from outside the tracker may establish incoming connections. I'd love some input on this part.

(It's true that suddenly requiring &private=1 for private magnet torrents would disrupt the workflows of private trackers who use magnets, but my guess is they would welcome the tradeoff of disruption for leak protection)

[the8472]

The private flag disables PEX, DHT, LSD, metadata exchange, multitracker extensions and any other alternative peer sources except incoming connections. So magnets and the private flag are mutually exclusive.

[AllSeeingEyeTolledEweSew]

Interesting. Where is it specified that the private flag disables metadata exchange? I couldn't find that in BEP 27 or BEP 9.

It does look like many clients today don't honor this, since I've been using private magnet links all day today. I haven't kept track of which clients do what, though.

[the8472]

utorrent, transmission, clients based on libttorrent, vuze

Have you checked whether the torrents actually contain the private flag?

[arvidn]

the private flag cannot be set (currently) until the metadata is received. So when a magnet link is just started, it won't be private. It will look for peers via the DHT, PEX and any tracker passed in through the link.

Once the metadata is received it will become private. It's possible that some aspect of the torrent mis this edge and keep allowing peer exchange for instance (iirc, libtorrent enables or disables PEX when the torrent is added).

In either case though, your IP and port is a critical secret for private torrents. As soon as you announce once you've lost. So I would say the utility of transitioning from non-private to private has probably only a marginal utility.

@AllSeeingEyeTolledEweSew what symptom do you see to conclude that many clients don't honor the private flag? I wouldn't expect anything to break if they do. After all, the torrents are still supposed to "work" once you have the .torrent file and you learn it's private.

again, the magnet link isn't private. The fact that it works is not because the private flag isn't honored, it's because the torrent isn't private (when added as a magnet link).

[the8472]

@arvidn my argument was more about the private flag disabling metadata exchange once it is known, which means magnets shouldn't be expected to work in the first place because all those who have torrents wouldn't be sharing the metadata.

[arvidn]

Ah, right. Of course.

[AllSeeingEyeTolledEweSew]

@arvidn: The symptom I see is that most magnet links posted by one private tracker I'm on actually do work, so some clients will transfer metadata despite the torrent being private. On some torrents I see I'm connected to peers but I never get metadata; I suppose these are all peers that implement the behavior @the8472 describes. Roughly, it looks like swarms with more than ~15 seeds will have at least one peer that will give me metadata.

@the8472: I spot-checked several infodicts to make sure they're marked private, they definitely are.

I guess it would be interesting to investigate which clients don't block metadata exchange. I'll investigate more and report back.

I'd like to understand the justification of blocking metadata exchange for private torrents though. Metadata exchange seems useful (takes burden off the tracker, whose hosting costs are often high). There's the downside I point out in the OP, of course, but it seems fixable.

[AllSeeingEyeTolledEweSew]

Or if there are good justifications I'm missing, I propose this behavior should go into a BEP. Otherwise evidently we're not all on the same page about the behavior!

[the8472]

The private flag is about the maximally conservative set of features to disable because private trackers don't like those features. Private tracker threatens to ban client due to feature X, users file issues, client makes X conditional on private flag not being set.

So if you want to use private torrents then it won't have to conform to whatever your usecase is but whatever the most paranoid bunch has decreed.

In the case of metadata exchange I guess the reasoning is that they make ratio-cheating easier since you don't even have to download the .torrent file which would show up in their logs.

[AllSeeingEyeTolledEweSew]

I would like to say that since clients still need to get peers from the tracker and the tracker logs announces, there's no loss of information to help ratio-cheating detection.

But since I've learned new things here, I'm not sure if I can represent the interests of private trackers for this discussion. I'm developing some projects tailored for private trackers so I've been in contact with some of their staff developers, but I am not one myself. I can try to track down opinions from staff, but their response times have no upper bound.

Are there links to previous discussion about the current behavior, of disabling metadata exchange? What parties were involved?

[the8472]

Are there links to previous discussion about the current behavior, of disabling metadata exchange? What parties were involved?

I don't remember any specific conversations, so I would have to some digging. So if you're interested I recommend that you do that.

I would like to say that since clients still need to get peers from the tracker and the tracker logs announces, there's no loss of information to help ratio-cheating detection.

Well yeah, my speculation is fairly tenuous reasoning. It would require a fairly specific attack model where someone knows an infohash and peers on that torrent (seedboxes?) but doesn't want to download the .torrent.

Maybe there was some other reason for it. Not everyone uses the private flag for ratio-tracking private trackers. Some are more concerned about leaking internal content to the public.

Anyway, if you want magnet links to work with private torrents that would require two changes. Adding it to the metadata exchange spec and convincing client developers and private tracker admins that this change of spec does not negatively affect their use-cases (I don't really want to call it security).

[AllSeeingEyeTolledEweSew]

FWIW, a brief survey shows that at least rakshasa libTorrent 0.13.x does ut_metadata transfers for private torrents.

[ckcr4lyf]

Was there ever any conclusion to this? Reading BEP0009, BEP0010 and BEP0027 there is no explicit mention that a torrent having the private flag set means a client should not advertise the presence of the ut_metadata extension.