internetarchive / dweb-archivecontroller

GNU Affero General Public License v3.0
7 stars 2 forks source link

IPFS - call to metadata/IDENTIFIER/FILE not currently supported #11

Open mitra42 opened 4 years ago

mitra42 commented 4 years ago

In ArchiveFile there is a call to metadata/IDENTIFIER/FILE, this is an addition to IA's standard metadata API, and in dweb-gateway led to file being added to a local IPFS super-peer and metadata returned with the IPFS hash.

This code has not (yet) been ported to the new ia-dweb repo (on github). It could be if someone has time, but has not so-far been high priority for reasons listed in the comment.

mitra42 commented 4 years ago

Why isn't fixing this anywhere near the top of the priority list :-)

a) the IPFS installer (ipfs-update) is down again see #10 b) we dont use IPFS for streaming files because there is no way to tell when a request is going to fail - we've been pointed at https://github.com/ipfs/interface-js-ipfs-core/issues/58 which has been high on IPFS's priority list since August 2016, and seems to have turned the simple issue of sending back an error on a likely failure into a really complex task that doesn't seem to be making progress. c) We can't push files into IPFS's DHT because it maxed out when we put 100k files in. Those files required us to have 200 threads submitting continues requests to update the DHT, and on request we disconnected, so files are ONLY available through our own peer. d) there is no peer-to-peer in browsers (or wasn't last time we asked), so every browser has to connect directly via WSS to our peer.

'c+d' together mean that our usage of IPFS isn't decentralized, and 'b' means we can only use it for files we know are small (icons).

So its really only a non-decentralized demo that we were using for the Icon files, which is not really of very high value.

Which is a long way of saying that this is unlikely to get prioritized anytime soon, but see next comment ....

mitra42 commented 4 years ago

if anyone wants to get it back in the library we'll be happy to stick it on a server - that would require