some bad code signatures for aarch64-darwin on cache.iog.io

[aryairani]

Describe the bug

Using haskell.nix and trusting the cache.iog.io substituter got me a build environment that was partially broken. Specifically, my haskell-language-server was dying on me with Killed: 9 and system log entries contained this entry complaining of an invalid code signature in libHSunliftio-0.2.25.0-H3rn7kmXhvKWwCKX2Y0ol-ghc9.2.8.dylib:

CODE SIGNING: process 15412[haskell-language]: rejecting invalid page at address 0x10da04000 from offset 0x50000 in file "/nix/store/4a0916lbmsryx1immcxmmwjvh01jy7d5-unliftio-lib-unliftio-0.2.25.0/lib/aarch64-osx-ghc-9.2.8/libHSunliftio-0.2.25.0-H3rn7kmXhvKWwCKX2Y0ol-ghc9.2.8.dylib" (cs_mtime:1.0 == mtime:1.0) (signed:1 validated:1 tainted:1 nx:0 wpmapped:0 dirty:0 depth:0)

default 15:25:28.320225-0400    kernel  CODE SIGNING: cs_invalid_page(0x10da04000): p=15412[haskell-language] final status 0x23000200, denying page sending SIGKILL

I confirmed that I only saw these crashes when I use the cache.iog.io substituter when building up my nix store. I searched my nix store for bad signatures and found several libraries with bad signatures, all of which I'd downloaded from cache.iog.org.

There are likely other improperly signed packages on the server that I'm not aware of.

• Build system aarch64-darwin
• GHC version 9.2.8
• Haskell.nix rev f629a8abac1bbb2168c1c763b9a80effef7156ea

Steps To Reproduce

Here is a bash script to download the packages I noticed issues with and confirm the bad signatures.

#!/bin/bash

while read pkgName; do
  host=https://cache.iog.io/
  narinfo=${host}${pkgName%%-*}.narinfo
  # get the relative URL to the zst file
  relativeZstUrl=$(curl -s "$narinfo" | grep '^URL:' | awk '{print $2}')
  zstUrl=${host}${relativeZstUrl}
  zstFile=$(basename "$zstUrl")
  # download the zst file, uncompress it, and verify the signatures
  curl -s "$zstUrl" | unzstd | nix-store --restore $pkgName
  for binary in `find $pkgName/ -perm -+x -type f`; do
    codesign -vv "./$binary"
  done
done << EOF
cw21sj1spq0wc56wprwiix1xlzybf2r0-HTTP-lib-HTTP-4000.4.1
fnngzx8z7v8aghlqav7v92rd3kblkk46-HTTP-lib-HTTP-4000.4.1
0z0j51pnnpmccbapilhjrpd605vz10rr-Cabal-lib-Cabal-3.10.1.0
qjw4vfmylna318dbcmnrcbq3bsfix0j2-githash-lib-githash-0.1.7.0
f438ms8yx2rjzaf6w0fm0hl7kbcvf250-hw-prim-lib-hw-prim-0.6.3.2
4a0916lbmsryx1immcxmmwjvh01jy7d5-unliftio-lib-unliftio-0.2.25.0
nf75rdjjj24qxrffvlbkymy2y3504pqv-libyaml-lib-libyaml-0.1.2
x6w39x5czd5fxgawn309ai3i5han6kl9-mono-traversable-lib-mono-traversable-1.0.15.3
zl7rzpykb556fbkdxfaaxws82njp7j7r-regex-tdfa-lib-regex-tdfa-1.3.2.1
7n13ddp1m3340ls4x3zx9qkjhqidi08x-zlib-1.2.13
hrpqpsjhw4yy82jwym31n87rrfv4yd39-comonad-lib-comonad-5.0.8
EOF

The first 9 report invalid signatures, the last 2 have valid signatures. I included a mix to help see the difference in output.

If you run the script, I recommend doing so from an empty directory to facilitate cleanup.

Here is the output of the script. The result of code signature verification is at the end of each line.

arya@halp nix-bug % ../verify-busted-zst.sh 2>&1 | grep valid
./cw21sj1spq0wc56wprwiix1xlzybf2r0-HTTP-lib-HTTP-4000.4.1/lib/aarch64-osx-ghc-9.2.8/libHSHTTP-4000.4.1-8FYwUMamrlY73fmwaF8u79-ghc9.2.8.dylib: invalid signature (code or signature have been modified)
./fnngzx8z7v8aghlqav7v92rd3kblkk46-HTTP-lib-HTTP-4000.4.1/lib/aarch64-osx-ghc-9.2.8/libHSHTTP-4000.4.1-8PZrx4V3FePGTHu7Myoffy-ghc9.2.8.dylib: invalid signature (code or signature have been modified)
./0z0j51pnnpmccbapilhjrpd605vz10rr-Cabal-lib-Cabal-3.10.1.0/lib/aarch64-osx-ghc-9.2.8/libHSCabal-3.10.1.0-6CI3UhnDiLqIrE6Z1d1Qqh-ghc9.2.8.dylib: invalid signature (code or signature have been modified)
./qjw4vfmylna318dbcmnrcbq3bsfix0j2-githash-lib-githash-0.1.7.0/lib/aarch64-osx-ghc-9.2.8/libHSgithash-0.1.7.0-GdlpKSIactcDD3g3hsuPwM-ghc9.2.8.dylib: invalid signature (code or signature have been modified)
./f438ms8yx2rjzaf6w0fm0hl7kbcvf250-hw-prim-lib-hw-prim-0.6.3.2/lib/aarch64-osx-ghc-9.2.8/libHShw-prim-0.6.3.2-LciqirBCrd75BM8aOM8Orv-ghc9.2.8.dylib: invalid signature (code or signature have been modified)
./4a0916lbmsryx1immcxmmwjvh01jy7d5-unliftio-lib-unliftio-0.2.25.0/lib/aarch64-osx-ghc-9.2.8/libHSunliftio-0.2.25.0-H3rn7kmXhvKWwCKX2Y0ol-ghc9.2.8.dylib: invalid signature (code or signature have been modified)
./nf75rdjjj24qxrffvlbkymy2y3504pqv-libyaml-lib-libyaml-0.1.2/lib/aarch64-osx-ghc-9.2.8/libHSlibyaml-0.1.2-DUQmlqcXcLvCo31KjqHrZo-ghc9.2.8.dylib: invalid signature (code or signature have been modified)
./x6w39x5czd5fxgawn309ai3i5han6kl9-mono-traversable-lib-mono-traversable-1.0.15.3/lib/aarch64-osx-ghc-9.2.8/libHSmono-traversable-1.0.15.3-Huqa1nXE1bVLwFSbqUBjT4-ghc9.2.8.dylib: invalid signature (code or signature have been modified)
./zl7rzpykb556fbkdxfaaxws82njp7j7r-regex-tdfa-lib-regex-tdfa-1.3.2.1/lib/aarch64-osx-ghc-9.2.8/libHSregex-tdfa-1.3.2.1-26rrbFfR9qrJjfKbftqY9s-ghc9.2.8.dylib: invalid signature (code or signature have been modified)
./7n13ddp1m3340ls4x3zx9qkjhqidi08x-zlib-1.2.13/lib/libz.dylib: valid on disk
./7n13ddp1m3340ls4x3zx9qkjhqidi08x-zlib-1.2.13/lib/libz.1.2.13.dylib: valid on disk
./7n13ddp1m3340ls4x3zx9qkjhqidi08x-zlib-1.2.13/lib/libz.1.dylib: valid on disk
./hrpqpsjhw4yy82jwym31n87rrfv4yd39-comonad-lib-comonad-5.0.8/lib/aarch64-osx-ghc-8.10.7/libHScomonad-5.0.8-d7V7K3MhgWDk2vXq6IMmE-ghc8.10.7.dylib: valid on disk

Expected behavior

Running codesign -vv on each binary should produce "valid on disk", not "invalid signature".

[aryairani]

Is there something I can do or provide to help get a fix worked out for this?

Removing the corrupted artifacts and rebuilding might be enough. You'll probably want to scan the whole store for corruption though, in addition to the ones I've identified; and I don't imagine that's something I can do directly myself, or I might try.

[angerman]

So what you are saying is that likely *-ghc9.2.8.dylib are busted. While *-ghc8.10.7.dylib are not. Now what I assume here happens is that stripping breaks them. The would need to be re-signed after stripping, (I believe nixpkgs did do this though).

[angerman]

According to @hamishmack we don't strip, so this must be something else.

I'll have to take a closer look at the derivations and the actual steps we do.

[aryairani]

Thanks @angerman. Let me know if there's anything I can do to help.

[aryairani]

I'm barely conversant in nix, but it's peculiar to me that the binaries are valid when I build them locally. (And the same binaries I build locally are valid on other machines.) It's only the outputs on cache.iog.io that are broken. I didn't think this sort of thing was possible with nix derivations, but here we are. 😅 Hopefully this is a clue.

[angerman]

Oh? That is indeed quite odd. Do note though that nix is only reproducible up to system level tools. And darwin is partially impure. We can't build darwin as pure a Linux (ala nixOS) and have to rely on some system tools. I do think we build codesign though. I'll keep a log of what I find out.

[aryairani]

Hi -- I wanted to check if there's any news or anything even I can do to help on this?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[aryairani]

Bump 🤜 🤛

[aryairani]

Am I the only one running on an arm64 mac? 😅

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[aryairani]

bump