parse .nar files on upload

[flokli]

This can be beneficial for all sorts of things:

• having some sanity checking on the uploaded .nar file
• assembling of a directory listing, which will allow partial substitution (only get one file out of a big .nar file)
• being able to feed the individual files to the chunker, instead of the whole .nar blob. Thus, always chunking on a file boundary, possibly getting better deduplication

If we do this and discard the initial .nar file, we need to be able to synthesize the original .nar file back, bit-by-bit.

[blaggacao]

Another one:

• more directly use the store as a data backend for a binary-registry-like use case

[flokli]

As blogged, I proposed a new store protocol, that has all metadata separated from the file contents, and provides chunking metadata. It's slowly making its way into Tvix.

I don't intend to pursue this idea here any further.

There will be a HTTP Binary Cache frontend for any tvix-store (called nar-bridge) at some point, so regular Nix substitutors can still talk to it, but to take advantage of most of the new protocol benefits you need to talk the protocol ;-)