search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
12.79k stars 1.52k forks source link

fetchTree unpacks tarballs incorrectly #11656

Closed typetetris closed 1 month ago

typetetris commented 1 month ago

Describe the bug

fetchTree {type="tarball"; url="https://csspeechstorage.blob.core.windows.net/drop/1.40.0/SpeechSDK-Linux-1.40.0.tar.gz";} doesn't correctly unpack the fetched tarball.

The tarball in question contains files not ordered by paths so files in different subpaths are mixed. It looks like only the last bunch of files for a subdirectory is kept.

Steps To Reproduce

/tmp via 🅶 via  v17.0.7 on   (eu-central-1) 
at 14:36:51 ❯ nix --extra-experimental-features fetch-tree repl
Nix 2.24.9
Type :? for help.
nix-repl>  fetchTree {type = "tarball"; url = "https://csspeechstorage.blob.core.windows.net/drop/1.40.0/SpeechSDK-Linux-1.40.0.tar.gz";}
{
  lastModified = 1723248714;
  lastModifiedDate = "20240810001154";
  narHash = "sha256-eC4Yl/zaVcEuVzKw1ZFuopftU4Y62o9QxgTTeO3v8rw=";
  outPath = "/nix/store/7y7k5xz1ryzzn41vvc5amz2fvziz4zjj-source";
}

nix-repl> :quit

/tmp via 🅶 via  v17.0.7 on   (eu-central-1) took 6s 
at 14:37:02 ❯ nix store prefetch-file --refresh --unpack https://csspeechstorage.blob.core.windows.net/drop/1.40.0/SpeechSDK-Linux-1.40.0.tar.gz
Downloaded 'https://csspeechstorage.blob.core.windows.net/drop/1.40.0/SpeechSDK-Linux-1.40.0.tar.gz' to '/nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz' (hash 'sha256-lbOKfzc+kybE4S3cFDe60DgwlgFaEY5VNzumBoUESCY=').

/tmp via 🅶 via  v17.0.7 on   (eu-central-1) took 16s 
at 14:37:24 ❯ diff -r /nix/store/7y7k5xz1ryzzn41vvc5amz2fvziz4zjj-source /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm32: libMicrosoft.CognitiveServices.Speech.core.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm32: libMicrosoft.CognitiveServices.Speech.extension.audio.sys.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm32: libMicrosoft.CognitiveServices.Speech.extension.codec.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm32: libMicrosoft.CognitiveServices.Speech.extension.kws.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm32: libMicrosoft.CognitiveServices.Speech.extension.lu.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm32: libMicrosoft.CognitiveServices.Speech.extension.mas.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm32: libpal_azure_c_shared_openssl3.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm32: libpal_azure_c_shared.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm64: libMicrosoft.CognitiveServices.Speech.core.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm64: libMicrosoft.CognitiveServices.Speech.extension.audio.sys.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm64: libMicrosoft.CognitiveServices.Speech.extension.codec.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm64: libMicrosoft.CognitiveServices.Speech.extension.kws.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm64: libMicrosoft.CognitiveServices.Speech.extension.lu.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm64: libMicrosoft.CognitiveServices.Speech.extension.mas.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm64: libpal_azure_c_shared_openssl3.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/arm64: libpal_azure_c_shared.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/x64: libMicrosoft.CognitiveServices.Speech.core.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/x64: libMicrosoft.CognitiveServices.Speech.extension.audio.sys.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/x64: libMicrosoft.CognitiveServices.Speech.extension.codec.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/x64: libMicrosoft.CognitiveServices.Speech.extension.lu.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/x64: libMicrosoft.CognitiveServices.Speech.extension.mas.so
Only in /nix/store/5a61v5ry9dkicsr5q13w6i22zd726zpw-SpeechSDK-Linux-1.40.0.tar.gz/lib/x64: libpal_azure_c_shared.so

/tmp via 🅶 via  v17.0.7 on   (eu-central-1) 
at 14:37:30 ❯ nix --version
nix (Nix) 2.24.9

Expected behavior

The tarball should be correctly unpacked, so that all files contained in the tarball are present in /nix/store/7y7k5xz1ryzzn41vvc5amz2fvziz4zjj-source or whatever its store path will then be.

nix-env --version output

at 14:37:51 ❯ nix-env --version
nix-env (Nix) 2.24.9

Additional context

I deleted ~/.cache/nix multiple times, while trying around with this. So the tarball cache was also refreshed multiple times. Also nix-collect-garbage -d got rid of the store path /nix/store/7y7k5xz1ryzzn41vvc5amz2fvziz4zjj-source multiple times for me, so hopefully it wasn't just a faulty download sticking to some cache ...

Priorities

This makes flake inputs of type tarball dangerous for now. So I guess it is pretty severe.

Add :+1: to issues you find important.

typetetris commented 1 month ago

Considering https://github.com/NixOS/nix/issues/10575 maybe the git tree object for a directory gets replaced when a tarball contains multiple files in different directories in mixed order. Would match to the observation, that only the files in the last mention of lib/x64 are present in the fetchTree result.

typetetris commented 1 month ago 
/tmp via 🅶 via  v17.0.7 on   (eu-central-1) 
at 14:47:19 ❯ tar -tvf SpeechSDK-Linux-1.40.0.tar.gz

...

-rw-rw-rw- root/root    251448 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/arm32/libpal_azure_c_shared.so
-rw-rw-rw- root/root    379184 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/arm64/libpal_azure_c_shared.so
-rw-rw-rw- root/root    379456 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/x64/libpal_azure_c_shared.so   # <--- MISSING
-rw-rw-rw- root/root    251448 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/arm32/libpal_azure_c_shared_openssl3.so
-rw-rw-rw- root/root    370992 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/arm64/libpal_azure_c_shared_openssl3.so
-rw-rw-rw- root/root    379456 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/x64/libpal_azure_c_shared_openssl3.so  # <-- NOT MISSING
-rw-rw-rw- root/root    469184 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/x64/libMicrosoft.CognitiveServices.Speech.extension.kws.so # <--- NOT MISSING
-rw-rw-rw- root/root   1689944 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/x64/libMicrosoft.CognitiveServices.Speech.extension.kws.ort.so # <--- NOT MISSING
-rw-rw-rw- root/root    202144 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/arm32/libMicrosoft.CognitiveServices.Speech.extension.kws.so
-rw-rw-rw- root/root    321504 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/arm64/libMicrosoft.CognitiveServices.Speech.extension.kws.so
-rw-rw-rw- root/root   1059940 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/arm32/libMicrosoft.CognitiveServices.Speech.extension.kws.ort.so
-rw-rw-rw- root/root   1717736 2024-08-10 02:11 SpeechSDK-Linux-1.40.0/lib/arm64/libMicrosoft.CognitiveServices.Speech.extension.kws.ort.so