can't update nixpkgs to the latest version

[bobbobbio]

When we try to update nixpkgs to the latest version, automation fails like this

these 2 paths will be fetched (1.18 MiB download, 6.99 MiB unpacked):
  /nix/store/17avbnwwl5r1r9x7fjridjfafdbgmhjd-bash-interactive-5.2p32
  /nix/store/7p41xclb67y3frgr1yyq52c1zi7iar69-bash-interactive-5.2p32-man
copying path '/nix/store/7p41xclb67y3frgr1yyq52c1zi7iar69-bash-interactive-5.2p32-man' from 'http://127.0.0.1:37515/'...
copying path '/nix/store/17avbnwwl5r1r9x7fjridjfafdbgmhjd-bash-interactive-5.2p32' from 'http://127.0.0.1:37515/'...
+ cargo build --release
   Compiling maelstrom-macro v0.12.0-dev (/home/runner/work/maelstrom/maelstrom/crates/maelstrom-macro)
   Compiling maelstrom-linux v0.12.0-dev (/home/runner/work/maelstrom/maelstrom/crates/maelstrom-linux)
   Compiling openssl-sys v0.9.103
error: failed to run custom build command for `openssl-sys v0.9.103`

Caused by:
  could not execute process `/home/runner/work/maelstrom/maelstrom/target/release/build/openssl-sys-527f7a6fc5f18479/build-script-main` (never executed)

Caused by:
  No such file or directory (os error 2)
warning: build failed, waiting for other jobs to finish...

Seems to have something to do wtih the openssl-sys package. I can't seem to reproduce it locally. @philiptaron claims that it does fail for him locally with nixos. I tried this out myself by running nixos in a docker image, but I didn't experience any failure.

In order to fix this we need to somehow be able to debug what is happening first.

[bobbobbio]

Here is some output from the nixos docker container showing it working

bash-5.2# git diff
diff --git a/flake.lock b/flake.lock
index 10f22bed..db343c4b 100644
--- a/flake.lock
+++ b/flake.lock
@@ -37,11 +37,11 @@
     },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1720255158,
-        "narHash": "sha256-rTB8Zd77oXk86JYQy6MoKFLFAohhzyN0FDPJWgeK930=",
+        "lastModified": 1725784592,
+        "narHash": "sha256-+q9YZ4HWL50epIkN7o25UmiPx9kHylkpwMwF2ynSdcc=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "d7b200a1978111e42d9e3fc8c0208d47b8304b29",
+        "rev": "17fb773308a8558abaa05b5df71cdaf3e5208e66",
         "type": "github"
       },
       "original": {
bash-5.2# scripts/run-tests-on-maelstrom-inner.sh
+ cargo build --release
   Compiling proc-macro2 v1.0.86
   Compiling unicode-ident v1.0.12
   Compiling libc v0.2.158
   Compiling autocfg v1.3.0
   Compiling version_check v0.9.5
   Compiling serde v1.0.210
   Compiling cfg-if v1.0.0
   Compiling log v0.4.22
   Compiling pin-project-lite v0.2.14
   Compiling once_cell v1.19.0
   Compiling byteorder v1.5.0
   Compiling bytes v1.7.1
   Compiling memchr v2.7.4
   Compiling itoa v1.0.11
   Compiling futures-core v0.3.30
   Compiling anyhow v1.0.87
   Compiling rustversion v1.0.17
   Compiling futures-io v0.3.30
   Compiling allocator-api2 v0.2.18
   Compiling futures-sink v0.3.30
   Compiling heck v0.5.0
   Compiling ident_case v1.0.1
   Compiling fnv v1.0.7
   Compiling strsim v0.11.1
   Compiling futures-task v0.3.30
   Compiling pin-utils v0.1.0
   Compiling futures-channel v0.3.30
   Compiling bitflags v2.6.0
   Compiling shlex v1.3.0
   Compiling equivalent v1.0.1
   Compiling rustix v0.38.36
   Compiling thiserror v1.0.63
   Compiling crossbeam-utils v0.8.20
   Compiling linux-raw-sys v0.4.14
   Compiling zerocopy v0.7.35
   Compiling camino v1.1.9
   Compiling typenum v1.17.0
   Compiling cc v1.1.18
   Compiling ahash v0.8.11
   Compiling generic-array v0.14.7
   Compiling utf8parse v0.2.2
   Compiling serde_json v1.0.128
   Compiling heck v0.4.1
   Compiling convert_case v0.4.0
   Compiling slab v0.4.9
   Compiling indexmap v1.9.3
   Compiling httparse v1.9.4
   Compiling syn v1.0.109
   Compiling http v0.2.12
   Compiling parking v2.2.1
   Compiling num-traits v0.2.19
   Compiling tracing-core v0.1.32
   Compiling powerfmt v0.2.0
   Compiling tower-service v0.3.3
   Compiling fastrand v2.1.1
   Compiling time-core v0.1.2
   Compiling num-conv v0.1.0
   Compiling tinyvec_macros v0.1.1
   Compiling tinyvec v1.8.0
   Compiling time-macros v0.2.18
   Compiling lock_api v0.4.12
   Compiling percent-encoding v2.3.1
   Compiling cpufeatures v0.2.14
   Compiling parking_lot_core v0.9.10
   Compiling try-lock v0.2.5
   Compiling num_threads v0.1.7
   Compiling quote v1.0.37
   Compiling want v0.3.1
   Compiling aho-corasick v1.1.3
   Compiling ryu v1.0.18
   Compiling atomic-waker v1.1.2
   Compiling signal-hook-registry v1.4.2
   Compiling mio v1.0.2
   Compiling socket2 v0.5.7
   Compiling getrandom v0.2.15
   Compiling httpdate v1.0.3
   Compiling unicode-bidi v0.3.15
   Compiling syn v2.0.77
   Compiling concurrent-queue v2.5.0
   Compiling slog v2.7.0
   Compiling regex-syntax v0.8.4
   Compiling piper v0.2.4
   Compiling event-listener v5.3.1
   Compiling dirs-sys-next v0.1.2
   Compiling futures-lite v2.3.0
   Compiling fs-err v2.11.0
   Compiling anstyle-parse v0.2.5
   Compiling is_terminal_polyfill v1.70.1
   Compiling event-listener-strategy v0.5.2
   Compiling http-body v0.4.6
   Compiling unicode-normalization v0.1.23
   Compiling async-task v4.7.1
   Compiling async-channel v2.3.1
   Compiling colorchoice v1.0.2
   Compiling anstyle v1.0.8
   Compiling anstyle-query v1.1.1
   Compiling async-lock v3.4.0
   Compiling dirs-next v2.0.0
   Compiling form_urlencoded v1.2.1
   Compiling thread_local v1.1.8
   Compiling anstream v0.6.15
   Compiling winnow v0.6.18
   Compiling slog-async v2.8.0
   Compiling clap_lex v0.7.2
   Compiling plain v0.2.3
   Compiling crypto-common v0.1.6
   Compiling block-buffer v0.10.4
   Compiling term v0.7.0
   Compiling digest v0.10.7
   Compiling idna v0.5.0
   Compiling crossbeam-channel v0.5.13
   Compiling rand_core v0.6.4
   Compiling is-terminal v0.4.13
   Compiling bstr v1.10.0
   Compiling take_mut v0.2.2
   Compiling bytesize v1.3.0
   Compiling pkg-config v0.3.30
   Compiling hex v0.4.3
   Compiling blocking v1.6.1
   Compiling vcpkg v0.2.15
   Compiling smallvec v1.13.2
   Compiling same-file v1.0.6
   Compiling async-fs v2.1.2
   Compiling glob v0.3.1
   Compiling url v2.5.2
   Compiling walkdir v2.5.0
   Compiling sha2 v0.10.8
   Compiling num_cpus v1.16.0
   Compiling async-walkdir v1.0.0
   Compiling fs2 v0.4.3
   Compiling either v1.13.0
   Compiling xdg v2.5.2
   Compiling openssl-probe v0.1.5
   Compiling terminal_size v0.3.0
   Compiling clap_builder v4.5.17
   Compiling openssl-sys v0.9.103 <======== look here it compiled just fine
   Compiling tempfile v3.12.0
   Compiling base64 v0.21.7
   ...

[bobbobbio]

I think I've figured out what was wrong. It was actually pretty simple in the end and I didn't notice this at first since I wasn't familiar with all the parts of the github action.

Basically, we have this cache of built Rust crates and the key for this cache doesn't include the nixpkgs version, so it was downloading stuff built with some older version of nixpkgs and it referred to files that weren't there. So basically I think the cache is the whole problem.

[philiptaron]

Link the PR / commit when you do fix it -- I'd love to see.

[bobbobbio]

I didn't fix it cleanly in one commit, but this is the one that did it 24abec8bea875bc59c7a0042430ae72c063ae8d3