Build Input Caching - Githubissues

mguentner commented 7 years ago

Beside distributing binary files, the build inputs of a derivation could be cached in IPFS. That would speed up CI quite a bit since the cache could move "closer" to the build instance (using an IPFS Gateway that is within the local network).

Current Status: @knupfer created fetchIPFS: https://github.com/NixOS/nix/issues/859#issuecomment-241787645 @CMCDragonkai is doing something similiar @ https://github.com/MatrixAI/Forge-Package-Archiving NAR replacement discussion: https://github.com/NixOS/nix/issues/1006

There are different methods on how to approach this:

put build inputs into IPFS and make a sha256:IPFS mapping inside nixpkgs (a huge set) Migration: easy (can be solved by a script that builds all /nix/store/ paths and puts them into IPFS) Support: every fetch derivation needs to be patched to look up the IPFS hash first, either fetch it from the local running daemon or a gateway. If no hash is found each derivation falls back to its normal operation (curl, git, svn etc.)
append the IPFS hash to each package (@knupfer's idea). Migration: takes a lot of effort (each derivation must be touched) Support: already implemented, however it must be added manually to each package

Should we produce an intermediate cache format (tar? -> can be chunked efficiently in IPFS) that is used to cache each src? This can then be used within each fetch derivation. As a big plus, the build inputs are deduplicated and share blocks with the official package. However it would be better just to reuse an already archived src (like tar.*, .zip etc.) and not to archive it again -> We need some form of manifest -> We need IPLD

CMCDragonkai commented 7 years ago

We are currently in the midst of going through ipfs internals (specifically libp2p) in order to create a haskell implementation and have a deeper integration of build input source hashes and ipfs architecture.

CMCDragonkai commented 7 years ago

https://github.com/MatrixAI/Forge-Package-Archiving/issues/1

CMCDragonkai commented 7 years ago

The biggest benefit is the ability to version srcs that are currently not versionable. Loads of packages distributors don't keep historical versions.

CMCDragonkai commented 7 years ago

We're halfway through implementing the haskell multiaddress library: https://github.com/MatrixAI/haskell-multiaddr (see improved branch). Still to do - binary encoding/decoding.

Perhaps we should meetup and compare notes?

mguentner commented 7 years ago

This script prints all tarballs for an expression:

nix-instantiate --eval --json --strict maintainers/scripts/find-tarballs.nix --arg expr '(import <nixpkgs> {}).hello'

there is also maintainers/scripts/copy-tarballs.pl that uses it to upload to S3

mguentner commented 7 years ago

First implementation: https://github.com/NixIPFS/nixipfs-scripts/blob/master/nixipfs/src/mirror_tarballs.py

NixIPFS / notes

Build Input Caching #1