Proposal: protocol scheme for urls

[RangerMauve]

Description

BitTorrent is great for sharing archives of static content through clients that download them to a folder, or render a single file from the folder in the case of videos. Magnet links have made the flow of creating a torrent and sharing it with people dead simple.

However, magnet links are limited in that they only use the search portion of a URL and don't allow you to specify a file or specify a hostname to navigate through. This is an essential feature for getting torrents to be loaded in web browsers and interoperating with their idea of what an "origin" is.

Prior / Existing art

IPFS is similar to BitTorrent in that it is a P2P network for transferring files. They developed their own ipfs:// and ipns:// protocols for use in browsers and link sharing. This is used in their various applications including ipfs-companion which enables people to view websites published on IPFS.

The bittorrent: protocol proposed on the ietf mailing list in 2010. It's an interesting start, but it ended up trying to do things like connecting to initial peers and wasn't as good as magnet links. (IMO) It also doesn't account for new features like BEP 46 which allows specifying a public key in a magnet URI instead of an infohash which allows people to publish mutable torrents on the DHT.

My Webrun project for which I created btih:// and btpk:// protocol schemes in order to load JS files from a torrent (from an infohash or public key). I modeled this after IPFS by having different schemes for the mutable and immutable torrents, but I'm not sure if this is the best way to go for users. I chose btih and btpk because that's what's used in the URNs for magnet links.

wtp-webext by @tom-james-watson is a Firefox extension that lets you load torrents as websites in a browser using WebTorrent and the experimental libdweb API in Firefox. He's using the wtp:// scheme to stand for web torrent protocol, and is planning on supporting DNS resolution for infohashes through the same way IPNS and Dat do it.

Design

The URL should support:

• Referencing infoHashes as the origin for the URL
• Supporting file paths into the torrent through the path of the URL
• Referencing public keys for BEP 46 mutable torrents
• Referencing domains that get resolved through DNS

Not sure if it's necessary to support all the features of a magnet URI like trackers. My reasoning is that a browser should primarily rely on a DHT / local discovery / PEX rather than trackers since it's more decentralized.

Protocol name suggestions

• bittorrent://
• bt://
• wtp://
• ???

---

I wanted to create this issue to get feedback on the idea and to figure out what would work. My goal is to have some sort of URL figured out by the summer time and to have at least a couple of applications that are able to torrents with it.

[gubatron]

bt:// 👍 (succinct)

[RangerMauve]

One thing to note is that public keys might be too long if encoded in hex since subdomains can only be 63 characters long. Dat kinda has this problem and IPFS is moving to base 36 encoding to mitigate it.

Using base36 would be useful, but I'm worried it'd be more effort for implementors since the infohash is hex and there's multiple kinds of base36 encodings out there.

[ssiloti]

I don't see why you would put the public key itself in a subdomain. BEP 44 items are looked up by the hash of the key (+ optional salt) so that hash would be the obvious choice for the subdomain.

[RangerMauve]

I think you still need the key for verifying the data you get from the hash.

At least, the magnet link contains the full public key and the reference implementation seemed to use it for verification. I might have misunderstood though.

Having just the hash and determining the type after doing a lookup would be ideal

[ssiloti]

The public key is included in the DHT response so it is not necessary to have it in the URL. BEP 46 includes the public key itself so that it can support salt. Off the top of my head I can't think of a good reason to use salt for web content.

[RangerMauve]

Perfect!

I thought that salt is useful if your public key is a sort of identity and the salt is for individual archives that you publish under that identity. Is the salt not part of the response? Maybe it could be part of the url as the username param? bt://salt@hash/file.txt

[ssiloti]

Salt is not part of the response because it is not necessary to verify the payload.

If you were going to include salt support I think including it as a subdomain of the pubkey derived domain makes more sense than using a username. Like I said though, it's not clear to me that this functionality is really needed.

[RangerMauve]

So something like bt://{salt}.{public key}/foo/bar.txt?

I guess we could defer the salt part for now. I'm not sure if anybody has actually used that feature for BEP 46 before anyway.

[ssiloti]

Oh, since you mentioned DNS length limits I thought we were talking about defining a mapping to http(s) URLs that resolve to a gateway. For URLs using our own scheme we don't need to constrain ourselves to the structure of http URLs, and certainly wouldn't need to have public keys in DNS names.

[RangerMauve]

I want to abide by the rules of common URLs like HTTP is to make it easy for people to reuse existing URL parsing libraries.

Mainly I want to the URL to be parsable by browsers, and I've had mixed results when using dat:// and ipfs:// URLs due to the length of the "hostname" portion.

[RangerMauve]

It seems that firefox's URL implementation parses new URL("bt://example.com") by putting example.com into the pathname Which is weird because Node handles it as you would expect by placing it in the hostname. I guess the browser implementation does extra magic based on the protocol. :/

[RangerMauve]

Kinda interested in this again. :P

[RangerMauve]

Put together a basic interface to talk to the bt:// protocol like you would HTTP.

https://github.com/RangerMauve/bt-fetch

Basically you can GET either a file, or append a / to get a list of files in that torrent.

The hostname portion of the URL is the 40 char infohash for the torrent. (mutable torrents are a TODO), and the pathname is the file you wish to load up.

I'm planning on adding this into the Agregore Browser later this week to see how it feels to talk to torrents using JS on a webpage. 😁

[RangerMauve]

It's working! Just loaded the Sintel video in a web browser using Agregore.

https://twitter.com/RangerMauve/status/1381400806356836353

Got a comment to use bittorrent:// for the protocol scheme. Maelstrom apparently used that in the past, but the docs for it seem sparse. I'll take a shot in the dark and see if it works.

Would a PR for a BEP be welcome?

[resession]

my vote is for bt. since nothing is fully accepted/standardized yet, bt makes more sense. simple, quicker, easier, also looks nice. it might look like a small deal, but more conviences for the average user the better, less typing is always better.