IPFS and package managers

[robcat]

In this note I will try to explore how the features of IPFS can be used in a typical package manager. In the following sections, a simplified description of how a typical current package manager is provided and contrasted with the features of a (hypothetical) ipfs-based package manager.

Package database

A package manager keeps a package database with basic info on all the available packages (i.e. metadata). Information on the package usually includes at least: name, version, cryptographic hash, list of dependencies. Periodically, the whole database is downloaded in compressed form and a local database is rebuilt.

With IPFS, a similar behavior can be obtained constructing one object per package, putting all the objects in a "folder", and pointing an appropriate IPNS entry to it. When the IPNS pointer changes the database is synced downloading only the upgraded package objects. Each database release can be linked to the previous one in order to form a chain and estabilish a total order. The dependencies can either be links to other ipfs objects (versioned packagees, dependency cycles not allowed) or plain strings (non-versioned, cycles allowed).

Package download and extraction

When the package manager needs a package file (i.e. the data), it uses the package name and version to request a compressed tarball from the preferred mirror (an ordered list of mirrors is configured by the user separately); delta upgrades are seldom used. Once downloaded, the data cryptographically verified and decompressed on the local filesystem.

With IPFS, the package file can be retrieved by CID with no need of specifying a mirror. The package data can be represented as a unixfs datastructure.

Package upgrade

In case of a package gets upgraded, the package manager redownloads the data from the mirror, performs the decompression and extraction steps, and overwrites all the old files.

With IPFS, the unixfs datastructure is updated in-place, with no need of redownloading the whole package.

IPFS benefits

• Lightweight nodes can afford to query ipfs instead of keeping a local database (saving disk space and processing power)
• Automatic delta upgrades (both of the package database and the packages)
• In case that the packages are distributed for multiple architectures (i.e. python2 and python3, x86 and x86_64, ...) many files could be deduplicated
• Easy integrity verification of packages, even after installation
• No need to maintain a static list of preferred mirrors
• Usage of local and/or transient IPFS nodes to accelerate the downloads (perfect when distributing a package to many nodes in a local network)
• The changes between two package versions can be easily listed

  Challenges

• PubSub #64 is needed in order to trigger the package database update
• How could package groups / metapackages be represented on ipfs datastructures?
• Is there a smart way to handle configuration files?
• Are there special considerations for source-based package managers?

[gavinmcdermott]

Hey @robcat, a quick note on the challenges and pubsub info:

• If you want to get a basic polling version running, a friend and I have a working version that we're using for sensors here
• PubSub discussions are progressing and have moved to this libp2p repo

[notpushkin]

Related: https://github.com/whyrusleeping/gx — a package manager over IPFS in Go

[davidak]

Are there special considerations for source-based package managers?

Of course the sources should be also in IPFS. We at NixOS have sometimes problems when source links disapear or change. Can also be used by multiple distributions when added the same way.