Status of projects integrating with dweb-transports.

[mitra42]

The following systems are integrated currently, updates are welcome.

[mitra42]

IPFS (from Protocol Labs) - integrates fully with the chunk write and read API's.

Lists and Key-Values are not implemented on IPFS, but are available over YJS which uses IPFS as its transport.

Problems: to be elaborated on, but are mostly due to:

• IPFS loses material occasionally - this looks like its a problem with Garbage Collection seeing urlstore items as collectable, should be fixed in js-ipfs#5765 but has not yet (Dec2018) made it into the release which is currently 0.31.7
• IPFS's DHT doesnt scale, Protocol labs are in internal discussions about how to do this - see notes#162, and until then IA material is only available to clients directly connecting to the IA's peer. This causes a Single Point of Failure issue.
• IPFS gives no error returns on failure, which in combination with the lost data, makes it unreliable, i.e. we have no practical way to determine if a stream we open to an IPFS resource will ever deliver data, so there is no way to fallback to WebTorrent or HTTP etc. There is a confusingly titled issue at interface-ipfs-core that addresses this issue of not being able to timeout a request.
• WebSocketStar (WSS) is required to enable clients to communicate with the server, and this has a Single Point of Failure.

Important Notes: Due to the combination of the lack of error returns and data loss, dweb.archive.org uses chunks, rather than streams for icons and prioritises WebTorrent over IPFS streams when both are available.

Due to the WSS and DHT single points of failure, we aren't currently considering IPFS for any anti-censorship applications.

[mitra42]

WEBTORRENT from Feross - integrates for streaming and chunking reads

Webtorrent is integrated for streaming files. We haven't yet implemented any writing mechanism. It does a nice fallback to HTTP if the file isnt available in WebTorrent. The majority of the Archive's files are available via Torrent and magnet links are included by default in metadata requested via the dweb.me gateway.

Issues: There is a single point of failure issue in that clients use a small number of known trackers, and can't use the BitTorrent DHT (due I believe to limitations in webRtc).

WebTorrent is our prefered

[mitra42]

GUN - to be written

[mitra42]

YJS - works, but not actively supported - implements KeyValue and Lists - details to be written

[mitra42]

HTTP

An HTTP version of all the APIs (chunk, list, keyvalue) has been integrated and is used as a backup and fallback.

Limitations:

• Monitoring is not implemented and would be very hard in client-initiated HTTP.
• It is, of-course, single point of failure by definition.

[rodneywitcher]

Wolk is working on an integration -- implementing chunk reads and writes as well as key-value solution