search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
11.49k stars 1.44k forks source link

Autodetect NAR compression #4534

Open edolstra opened 3 years ago

edolstra commented 3 years ago

Followup to #4464. Re-uploading a NAR with a different compression method invalidates the (cached) .narinfo, so it's better if we get rid of the Compression field and autodetect the compression method.

zimbatm commented 3 years ago

It might be helpful to upload the NAR files with the Content-Encoding header set, for that purpose. That way it's possible to query the compression without downloading the file and looking for magic headers. Or use the Content-Type. S3 supports recording both of those (as does Google Storage).

edolstra commented 3 years ago

That would only work for HTTP, but we shouldn't need it since the first few bytes allow the compression method to be detected. Hopefully we can use libarchive for this (see #3333).

flokli commented 3 years ago

Brotli seems to have no standardized framing format, according to https://github.com/google/brotli/issues/298 and the followup questions.

I wonder if we still want to keep support for uploading, and if so, strongly discourage using it in places where we can't set the Content-Type to facilitate the format detection.

stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info