IPFS support

[andrew]

Motivation

Opening an issue to discuss different ways that IPFS could be used to enhance wapm.

Proposed solutions

There's a lot of different levels of integration with IPFS that could be undertaken, summarizing three of the main ones here:

1. Storage and transport of package contents

When a package is published to wasm.io, along side storing the package on the server, the registry could also add the package tarball to IPFS and store it's Content Identifier (CID) in the database.

Then when the client queries the graphql API it'll get back both a http url to the package and a IPFS CID, so it can download the package from the IPFS swarm of connected peers.

Making wapm content addressable also bring integrity checking along for free, as IPFS ensures that the content it has downloaded is what was requested, much like sha256 checksums in many existing package managers.

There's also been some work done in getting go-ipfs compiled to WASM: https://github.com/ipfs/go-ipfs/issues/5694 and https://github.com/ipfs/go-ipfs/pull/5878 which would mean that the IPFS client can be bundled, distributed and updated via wapm itself!

2. Read-only Package metadata and indexing

Having package contents on IPFS is great but it still requires querying the centralized wasm.io graphql server over http, the next step would be making versioned metadata about the packages available via IPFS as well.

There's a number of different approaches that could be done with this, a certain amount depends on internal implementation details of the registry.

In general snapshotting the metadata about a package and it's versions for each release in a deterministic way would make it possible for IPFS to query wasm.io either via DNSLink or IPNS for a root CID to a tree of package metadata, which would be updated every time a new version of a package is published.

Publishing packages would still be done by using the traditional authenticating and uploading to the centralized wasm.io registry.

3. Decentralized publishing

Making a full decentralized package manager that has human readable package names and some levels of trust is much more involved and requires some form of Decentralized Identity which gets into blockchains and other fun things, likely behind the scope of the initial integration but there's a lot to explore here

Additional context

The IPFS package managers repository: https://github.com/ipfs/package-managers/

Also see @warpfork's flowchart of integration levels: https://github.com/ipfs/package-managers/issues/50

[xmclark]

@andrew Thanks for reporting this. I like this description because it gives an incremental path forward for centralized package managers like wapm.

At this point, I think we are interested in evaluating ipfs for "Storage and transport of package contents".

I have studied both gx and npm-on-ipfs. Unless I'm reading it wrong, I understand that both package managers choose to not manage ipfs installations or local ipfs repo. Instead, they require the user to install go-ipfs and ipfs init their local repository.. The package manager simply executes the correct ipfs shell command during install or publish. Is this the preferred strategy?

I like that approach because then our tool does not need to manage an ipfs repository or an installation. The user experience is degraded though, as we now require the user to do some extra work to get ipfs "turned on". That can be mitigated with good UX and documentation.

[andrew]

npm-on-ipfs does actually run and manage a local in process ipfs installation by default using js-ipfs but has the option to either start a separate go or js ipfs daemon or to connect to an already running one, basically every option available!

gx doesn't start or setup an IPFS node for you but if it can't find one running locally it will use the public ipfs gateway instead (detection code here), although loading from the public gateway will negate most of the benefits of IPFS!

Another example is qri which handles starting and setting up a the ipfs node transparently to the user with commands like qri setup and qri connect.

One option may be for wapm to try and detect if IPFS is running on it's standard port, if use that via the local http api for downloading tarballs along with proving command line flags for passing a custom address for an IPFS node. That way if a user is has ipfs desktop or go-ipfs running things should just happen automaticlly.

You could also go further and look to see if there is already a .ipfs repository in a standard location, if so attempt to start an ipfs daemon and then use that.

On the registry side, ipfs-cluster would be a great option for ensuring there's lots of peers available for downloading packages from.

[MarkMcCaskey]

@xmclark We could have a config value for ipfs address so the user doesn't have to pass it each time.