How to work around upstream archives being removed

[akirak]

As reported in https://github.com/emacs-twist/twist.nix/issues/159#issuecomment-2135084868, a configuration fails to build if one of the source archives tracked in archive.lock is removed from the original location. Using archives is generally safer for packages on GNU ELPA, as some packages on the registry are built in specific ways.

Apparently, when a package on GNU ELPA is updated, the previous version is compressed with lzip, and the initial uncompressed archive is removed. This breaks reproducibility of this project. Also, .tar.lz is not natively supported by Nix flakes, so it wouldn't fit the current system of archive.lock anyway.

I don't come up with a solution right now. Perhaps some extra infrastructure will be needed? I wouldn't be happy with it, but the issue needs to be addressed to make this project usable.

[terlar]

It seems you are right regarding builtins.fetchTree not supporting .tar.lz.

I guess another way would be to use a derivation, at least that seems supported: https://github.com/NixOS/nixpkgs/blob/master/pkgs/by-name/mo/moe/package.nix

It seems that the stdenv.mkDerivation will handle a source with .tar.lz as long as the nativeBuildInputs including lzip are added. But I guess that won't really fit with the rest of the current model?

[terlar]

Actually it seems to work when I test on my machine:

$ nix build --eval-store /tmp/test-store --expr 'builtins.fetchTree { type = "tarball"; narHash =
 "sha256-WGMPCLj/RH8we9HEGTxvyc9wyG70H+IRNq7p0G+IcjE="; url = "https://elpa.gnu.org/packages/csv-
mode-1.24.tar.lz"; }'
$ grep -r 'Version: 1.24' /tmp/test-store/nix/store/dcpxq8ysmxpvsbhwvpzv6amk9hbdvk94-source
/tmp/test-store/nix/store/dcpxq8ysmxpvsbhwvpzv6amk9hbdvk94-source/csv-mode.el:;; Version: 1.24

[terlar]

Now the question is, how do we know when to go for the .tar and when to go for the .tar.lz. I did test with a builtins.tryEval. But it is not possible to catch the failure from builtins.fetchTree. I guess since you already have the information from gnu-elpa we can do something with that.

[akirak]

Actually it seems to work when I test on my machine:

Confirmed on Nix 2.18.2. Thank you for providing the information. The lzip support is undocumented, but it would be safe to assume the functionality in all future versions of Nix.

Now the question is, how do we know when to go for the .tar and when to go for the .tar.lz

narHash doesn't change no matter if the tarball is compressed or not. This means you can fetch the compressed version by simply suffixing the URL with .lz once a package becomes outdated on GNU ELPA. Using some scripting language, it will be possible to verify all URLs in archive.lock and replace all non-existent URLs with their corresponding .lz archives. You can solve the missing archive issue by adding the step on your CI, right?

since you already have the information from gnu-elpa we can do something with that.

The above rewrite could be done in twist if Nix were run in impure mode. It would be possible to define a new mode of operation to fetch and parse the latest version of the archive contents (as done in the current lock operation), compare the versions of each tracked package, and update them if necessary. Unfortunately, I don't think this can be done as part of the pure build process with Nix.

I am not totally happy with this workaround. It would be nice if GNU ELPA provided the compressed format for all versions of all packages. package.el still requires the uncompressed format, but any other third-party tool can benefit from getting permanent URLs. It may worth asking them.

[terlar]

Yeah, I agree it might be worth requesting this. I can understand not saving the uncompressed version, but I don't see a reason why they couldn't include the compressed version from the start (except that it cannot be directly used, but that goes for all previous versions as well). That would make things easier on our end.