Proof of concept: Webtorrent

[eklem]

Start figuring out webtorrent as a way to syncing data.

[eklem]

Stuff to figure out:

Seeding, creating, downloading

• How to seed JSON objects and not files.
• After creating a torrent and seeding: second time you open the browser, how to continue just seeding.
• If you create a new webtorrent for each time new data is created: In the downloading end, is it possible to copy in existing data and not download those parts? I guess it's some hashing verifying existing content.

Trackers:

• Which tracker is used?
• How to set up a tracker?
• Can access to torrents on the tracker be based on user authentication?

[eklem]

Reasons to use Webtorrent:

• Mature technology
• Seems okay simple
• Works well in the browser
• Tracker infrastructure exists
• Trackers also seems simple enough

Drawbacks:

• Per user access
• Have to figure out how to make it fit the business plan. Maybe it needs some changes because of how trackers works

[eklem]

I guess the business plan will be geared more towards either public search engines or private for you only. Proof of concept will be private, running only in your (one) browser, and then it will be public if you want to sync it.

[eklem]

So, the development plan can be to first offer:

• offline search engine only living in your browser
• public search engines through webtorrent

For the webtorrent part theese are inherently read-only. To fix this there are many small steps that can be taken:

1. Read only. You create a public search engine and that's it.
2. Each time you update a new webtorrent is created. Everything is re-indexed.
3. Only index new & changed parts.
4. Create incremental backup-like schema where updates are separate webtorrents. Every update during a day is a new ..-release. Every day is a new .-release (containing all new ..-release) and every week is a full new release (containing everything).
5. See if it is possible to copy over old (and identical files) from old webtorrent to new webtorrent. This way only new stuff will be downloaded. Then you could have the backup schema inside the webtorrent. This would replace 3 and 4, except for the part with indexing only new stuff.

Business plan:

1. Start with public an private (offline/local only)
2. When interest and resources, try to develop authentication and authorization into the tracker software. Should be open spireuso people can set up their own trackers. Could be a joint effort wit the people behind the Webtorrent-technology? Not sure how difficult. Make it possible to integrate with i.e. Auth0 and some open source alternative.

[eklem]

Private torrents

Quick wins that will make you have the seed(?) hash to get to use a torrent. If the tracker is also behind login, it's a first low level security solution:

• An issue on private torrents for Webtorrent tracker.
• PR on better documenting private torrents + code fixes + tests.

Possible real secure solution:

But if you want to check if user is allowed to access this torrent, you can maybe do as suggested in the second answer to this question:

I understood that I can use a private tracker and seed files privatly with these options. But is there any additional option on client side to hook in if someone requests one of my seeded files? This would make it possible to implement some security logic on the client, not on a centralized tracker, which could be untrustable from a client's perspective. In addition, you could authorize requests even with using dht.

There is an extention API for the Bittorrent protocol and a Metadata example extention.

How to intercept:

torrent.on('wire', function (wire) {
  wire.use(myExtension())
})

[eklem]

Files seems to just be data, so both the ones that creates and seeds a torrent and the ones downloading are just keeping this in memory. How to make this persist?

Figuring this out will also maybe make the incremental backup possible to work. Where a new torrent download takes copies of existing files into memory so you don't have to download. But it will need to happen after initiating download and before actually downloading. Or maybe start, pause, copy and re-start.

[eklem]

Persist webtorrent in the browser: Done by SilentBot1 on this issue: Save a complete Torrent with its Files for later reuse!

[eklem]

Private torrents take

• Encrypt all webtorrents
• When creating webtorrent: dht: false
• When seeding file, set private: true

So there will be no way of finding out which torrents are seeded, you need to know the hash for it. In addition, you need to know the key. This can be stored together with the hash at i.e. Auth0 together with .

This way it will be easier to create:

1. private torrents, only for you
2. private torrents, for many
3. public torrents. These will not be encrypted.

For number 2 and 3, we need to figure out how to make them available to others.

2. You share a search engine with another user: "Add user" and add email address of their user. They will get an extra search engine (managed by you) in their user interface.
3. Subdomain? That's maybe too much work, and maybe for users that are almost creating their own search engine outside Nowsearch.xyz. Set it as public and have the url nowsearch.xyz/public#hashToWebtorrent which is also the ID for the search engine?

For super private stuff, you set up your own tracker or we can also offer access accept/deny on this level:

torrent.on('wire', function (wire) {
  wire.use(myExtension())
})