Proposed Federation Design #1

[ReisenB]

Presented below is a rough sketch of what a decentralized bittorrent tracker system could look like. The system is built primarily on IPFS, and is in places predicated on js-ipfs being usable (though a cut-down version could still work without it). The system is composed of three servers and a frontend. The first two servers can be easily forked at any time, and the third can be rebuilt on-demand from the information provided by the former two. These server and frontend packages could all run on a single physical machine, or they could run as a federation (the better scenario). The first two servers do represent points of centralization of authority, but the actual content is fully distributed. The third server and frontend will ideally be runnable entirely within the browser of the end user, meaning the surface area for debilitating attacks on the system is limited entirely to temporarily disabling the creation of new user profiles, or (in a much harder to carry out attack) temporarily freezing the upload of new files by specific groups of users.

[ReisenB]

Feed Server

Purpose

Receive uploads directly from users, and patch them into the IPFS feed for that user.

Description

This server maintains a separate IPFS feed for each registered user. Users may upload torrent files (or perhaps anything) directly to this server. On receiving a file, the server will add it to the end of that user's associated IPFS feed. The feed takes the form of a linked list (or blockchain if you like). This server may also generate an RSS feed of the last ~100 or so items in the IPFS feed for direct use in torrent clients.

Language

Anything with an IPFS API would work. Currently I am only confident that go-ipfs-api and js-ipfs-api are fully supported, but have not examined the state of the others in detail. Node would be nice since that provides possibilities for a feed server to be run from a browser. It shouldn't be too demanding in terms of resource requirements, at any rate. The only actually demanding component in any of this is the search server.

Formats

The format of the feed doesn't matter too much, but JSON seems reasonable. The IPFS DAG format should also be examined and understood. Sample JSON follows:

{last_item: '/ipfs/QmLastItemInFeed', item: {filename: '[>Commie] Your Least Favorite Animu.mkv', magnet: 'magnet://btih:etc.', categories: ['anime', 'eng-subtitles', 'jp-audio']}, rss: '/ipfs/QmLatestRSSFeed/latest100.rss'}

Note: for the RSS feed to be directly readable by standard clients perhaps it'd be sensible to instead place it as a file in the feed's root along with the above JSON. i.e. /ipns/QmFeedPubkey/item.json and /ipns/QmFeedPubkey/latest100.rss

[ReisenB]

Index Server

Purpose

Collate disparate IPFS feeds generated by one or more feed servers into a coherent structure.

Description

This server gathers together IPFS feeds as generated by feed servers. To use, uploaders would register their feed pubkey with this server along with some chosen username etc. This server would also handle things like 'trusted' statuses and so on per user, if that's what the index operator wants to do (not recommended tbh). This server can be forked at any time by resolving the current IPNS address and applying that hash as the current state of the new server.

Language

Again, anything with an IPFS API would work.

Formats

Again either ordinary JSON or perhaps the IPFS DAG format (which can be rendered as JSON anyway).

{last_index_state: '/ipfs/QmHashOfTheLastVersionOfThisJSON', registered_users: [username: '>Commie', feed: '/ipns/QmUserFeedPubkey', in_cartel: true], site_css: '/ipfs/QmHash/styles.css', site_name: 'Not-Nyaa'}

[ReisenB]

Search Server

Purpose

This server monitors the index produced by an index server and creates a searchable database of all currently linked items.

Description

The search server allows for the search of all feeds associated with a particular index by monitoring said index. The server will periodically poll all associated IPFS feeds for new releases, and add all found items to an internal SQL database. Users can send standard xmlhttprequests to this server and search this DB by title, category, upload date, uploader etc. This component can be accessed through ordinary clearnet channels with a domain name and so on, or potentially through IPFS directly through a custom IPFS service (assuming that works correctly). It is intended that eventually the end user will have the option to pull in a local copy of this component to their HTML5 local storage.

Language

This component is eventually intended to be runnable on both a remote server and in the browser locally. To this end, browserifiable Node code would be most convenient. The caveat here would be whether or not the one-thread-wonder could keep up with it. Actual testing would probably be required to determine this. Alternative arrangement would be to do two implementations, perhaps one in Go and a second in JS.

Formats

This server accepts queries regarding the known state of an index and returns the appropriate item(s), probably as JSON since it's already being used for everything else.

[ReisenB]

Frontend

Purpose

The frontend system connects to a search server and renders out an actual HTML page based on the user request.

Description

This component comprises the end-user hook for actually viewing the contents of the federation. It should connect to a search server, be it remote or local, and query it for things like (for instance) a listing of the most recently uploaded files known by the search server, a listing of the uploads by a given uploader, and a listing of torrents that match a given keyword or set of keywords. The frontend should then take those results and render them out as HTML. It should take style cues as specified by the index server that the search server is monitoring.

Language

Node. This part must run in the browser, and it'd be nice if it could also run on a server so that users who don't like JS can still use the site.

Formats

Renders out HTML. A sample request to the JS API might look something like getPageHTML('/ipns/QmPubkeyOfSearchServerEitherRemoteOrLocal', '/ipns/QmPubkeyOfIndexServer', '# /<uploader-id>/view/<torrent-id>', (html, err) => {document.body.innerHTML = html});