Trouble building eclipse 3.5.2 in old nixpkgs from 2010 (aside: referencing "expired" binary caches)

[deliciouslytyped]

I didn't think I'd end up running into https://github.com/NixOS/nixpkgs/issues/55588, https://github.com/NixOS/nixpkgs/issues/53653 , but the recently released ghidra wants a really old version of eclipse for its ghidraDev plugin (edit: misunderstood an error). I figured I could try getting an old version of eclipse from an old version of nixpkgs, since (sadly imo) nixpkgs only really accepts one version of a package at a time.

Building off an old github release like the following yields the pasted errors:

nix-shell -I "nixpkgs=https://github.com/NixOS/nixpkgs/archive/d75d0cbb6a351fa6ccbdcfe4abe96c8c582ea968.tar.gz" -E "(import <nixpkgs> {}).eclipse" -v 2>&1 | xclip -i -selection clipboard

these derivations will be built:
[...snip...]
  /nix/store/vg4hyqa8jgwqpz966r728yxcfn76frkw-Eclipse.drv
building '/nix/store/zmw16913cabf8cvbdg2kxhax801ap5ic-bootstrap-tools.cpio.bz2.drv'...
curl 7.19.4 (i686-pc-linux-gnulibc1) libcurl/7.19.4
Protocols: tftp ftp telnet dict http file
Features: IPv6 Largefile
downloading /nix/store/nh1xjqkkccglnickb0hlkdznvbn61yjx-bootstrap-tools.cpio.bz2 from http://nixos.org/tarballs/stdenv-linux/x86_64/r16022/bootstrap-tools.cpio.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
143   286  143   286    0     0   3916      0 --:--:-- --:--:-- --:--:--  8411
curl: (1) Protocol https not supported or disabled in libcurl
builder for '/nix/store/zmw16913cabf8cvbdg2kxhax801ap5ic-bootstrap-tools.cpio.bz2.drv' failed with exit code 1
cannot build derivation '/nix/store/k3x2hhdqjpqavrjr89h12g8536fibyq5-bootstrap-tools.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/k858rhkrp20d2x25kz3dq5z0b2zxsnzw-bash-4.1-p2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/1ms1c0i34nlkv7hrr8frlmyk3ark9sb6-hook.drv', '/nix/store/25q38m484fcwbiaa2z3vx91zsf15s8gy-fontconfig-2.7.3.drv', '/nix/store/3266y6lpddav8pgynwng80mlfhh1v21h-zlib-1.2.3.drv', '/nix/store/3lbb3frzrl08p3gphy4dnpcqrjzz2wxw-gtk+-2.16.2.drv', '/nix/store/3z6wbfx6nncsxvf6z2jk6jbr587wlzb4-freetype-2.3.11.drv', '/nix/store/42vn11x75cli2lk02qwr3gnznnnzysrv-glibc-2.11.1.drv', '/nix/store/638lf3p857cjmkyj8jzn5l28lqv9389l-libXtst-1.1.0.drv', '/nix/store/7w995gf6nkid8vbwdfqxhjk1qk2rc22k-stdenv-linux.drv', '/nix/store/9xb2b10imbc7g75s555m0kj2l0wami95-jre-1.6.0_16.drv', '/nix/store/bpgnsjgp9aiwdr99ymw40g2gry3r6wc1-eclipse-SDK-3.5.2-linux-gtk-x86_64.tar.gz.drv', '/nix/store/c45hxigp6ajiyd0pj1l3s77mzbnbaxp0-libXrender-0.9.5.drv', '/nix/store/g0m5lwcr6sq7pmkraias6w7ws0f44vvm-bash-4.1-p2.drv', '/nix/store/k858rhkrp20d2x25kz3dq5z0b2zxsnzw-bash-4.1-p2.drv', '/nix/store/li1035n6b2k1ph26ajxq3dkvh9jp23id-libX11-1.3.2.drv', '/nix/store/rd6ibbivnh6hhhncl7w56n73asc9s5x6-patchelf-0.5.drv', '/nix/store/snsnysh14b3qy17dp9v3cqki8zrk3b34-glib-2.20.1.drv', '/nix/store/vg4hyqa8jgwqpz966r728yxcfn76frkw-Eclipse.drv' failed

By @LnL7 's advice, I used nix-store --add-fixed sha256 <path> (which is now documented with https://github.com/NixOS/nix/pull/2724) to add that archive, and other archives that were missing with later similar errors:

trying http://ftpmirror.gnu.org/findutils/findutils-4.4.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
error checking the existence of http://nixos.org/tarballs/sha256/24b9a850af45e1a277e234b9eb090b52305a2e1c6b02addeb3ae98b4b49d37ce:
curl: (1) Protocol https not supported or disabled in libcurl

There's an HTTPS redirect there which is the original problem, but if you try to resolve the tarballs url in the browser you also run into a 404.

edit: correction about ghidraDev wanting an old eclipse, I misunderstood an error

[LnL7]

This is causing the problem, http://nixos.org/tarballs redirects to https before the tarballs.nixos.org redirect resulting in a https request while the bootstrap build can't fetch that yet.

https://github.com/NixOS/nixos-org-configurations/blob/517cabda83b1e0678e39e4dd8c221308e547d8ca/nixos-org/webserver.nix#L155-L157

/cc @grahamc

[jtojnar]

since (sadly imo) nixpkgs only really accepts one version of a package at a time

This is not true. We are fine with multiple versions of single package, as long as there is good enough reason to keep the old one. Most of the time there is not any, so the concerns about maintenance costs of extra packages and security issues of ancient code prevail.

[deliciouslytyped]

(Yeah, my bad, I knew better and yet I still said that. I'm tired.

I would like it if version hopping across already created scripts wasn't possibly only through the git history, but I can understand that there would be problems with that under the current model. I would still like it if it was possible eventually.)

[7c6f434c]

1. Keeping all version in-tree would require some kind of policy for choosing the moment we give up on updating the deps.

2. Maybe if it is a problem only with stdenv, we should just document how to get the list of things to fetch, and how to prefetch them using a fresh Nixpkgs checkout?

[deliciouslytyped]

@7c6f434c To be fair, currently the problem seems to be a matter of the old sha256 interface having been removed or somesuch / the cache being reorganized. Though in the theoretical case that sha256 is insecure I'm not sure what to do about that.

I'd be quite happy if it was just a documentation problem, if it's a solution that's still reasonably viable for a user.

However currently what I did, was I had to run through the build process numerous times to collect the names of the failing fetches, and then google for the archives, and add them using --add-fixed

[deliciouslytyped]

I've run into a new failure while building util-linux-ng chmod: changing permissions of '/nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/bin/mount': Operation not permitted in

make[2]: Entering directory `/build/util-linux-ng-2.18/mount'
building install-binPROGRAMS
test -z "/nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/bin" || /nix/store/7w8b9g33z1vffv6y2swr3dbiddsrysdl-coreutils-8.7/bin/mkdir -p "/nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/bin"
  /bin/sh ../libtool   --mode=install /nix/store/7w8b9g33z1vffv6y2swr3dbiddsrysdl-coreutils-8.7/bin/install -c mount umount '/nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/bin'
libtool: install: /nix/store/7w8b9g33z1vffv6y2swr3dbiddsrysdl-coreutils-8.7/bin/install -c .libs/mount /nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/bin/mount
138: Trace output                                    ok
libtool: install: /nix/store/7w8b9g33z1vffv6y2swr3dbiddsrysdl-coreutils-8.7/bin/install -c .libs/umount /nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/bin/umount
building install-sbinPROGRAMS
test -z "/nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/sbin" || /nix/store/7w8b9g33z1vffv6y2swr3dbiddsrysdl-coreutils-8.7/bin/mkdir -p "/nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/sbin"
  /bin/sh ../libtool   --mode=install /nix/store/7w8b9g33z1vffv6y2swr3dbiddsrysdl-coreutils-8.7/bin/install -c losetup swapon '/nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/sbin'
libtool: install: /nix/store/7w8b9g33z1vffv6y2swr3dbiddsrysdl-coreutils-8.7/bin/install -c losetup /nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/sbin/losetup
libtool: install: /nix/store/7w8b9g33z1vffv6y2swr3dbiddsrysdl-coreutils-8.7/bin/install -c .libs/swapon /nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/sbin/swapon
building install-exec-am
make  install-exec-hook
make[3]: Entering directory `/build/util-linux-ng-2.18/mount'
building install-exec-hook
cd /nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/sbin && ln -sf swapon swapoff
chmod 4755 /nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/bin/mount
chmod: changing permissions of `/nix/store/wg33bxhbdnb9hl6asn6gxwrnzfjhdv3w-util-linux-ng-2.18/bin/mount': Operation not permitted
make[3]: *** [install-exec-hook] Error 1
make[3]: Leaving directory `/build/util-linux-ng-2.18/mount'
make[2]: *** [install-exec-am] Error 2
make[2]: Leaving directory `/build/util-linux-ng-2.18/mount'
make[1]: *** [install-am] Error 2
make[1]: Leaving directory `/build/util-linux-ng-2.18/mount'
make: *** [install-recursive] Error 1

builder for '/nix/store/86jim6yp6kj3p6rqq7apqablida8q094-util-linux-ng-2.18.drv' failed with exit code 2

[7c6f434c]

@7c6f434c To be fair, currently the problem seems to be a matter of the old sha256 interface having been removed or somesuch / the cache being reorganized. Though in the theoretical case that sha256 is insecure I'm not sure what to do about that.

Arguably, there are multiple issues.

0. It would be nice to have a published policy for retention/resource management in binary cache, as well as in tarball cache. I frankly have no idea if failure to grab from binary cache is a misconfiguration on your part, a weird effect of long-term compatibility or just the file having been finally garbage-collected. The same question but separately — about tarball cache.

1. One could try to fetch from upstream URLs; even if they were already HTTPS and unaccessible to bootstrap curl, we could write a script to export a list of urls and just prefetch them.

1a. We do not always write the list of URLs correctly, i.e. in a way that takes into account upstream policies on moving old sources to a slower archive mirror.

2. Even if tarball cache URLs have changed, exporting them, auto-rewriting and prefetching might still work.

2a. If sha256 is insecure and sha256-addressing is retired, it is possible to have a script with all correspondences between old SHA256 URLs and whatever new scheme for tarballs is supported (for tarballs that are retained, see point 0).

[deliciouslytyped]

I think we should separate this out into an appropriately named issue, or rename the current one, do you have any ideas?

[7c6f434c]

I thought your plan for the current issue was to discuss and hopefully work around all the issues arising from zero to Eclipse. I also replied by mail without checking for updates, sorry.

What is your current preferred use/goals for this issue?

[deliciouslytyped]

Yeah you're right but this more general than my specific problem, but we can continue here, that's fine for me.

[7c6f434c]

Well, I can only hope there are not additional fetching problems to arise… Otherwise I hoped this could be useful for triage, too. If I decide to continue in general, I will create specific issues.

[deliciouslytyped]

(In hindsight, I have no idea why I thought sha256 is being deprecated for sha512...is it?)

[7c6f434c]

Not actively yet.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.