search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
12.39k stars 1.49k forks source link

FOD build with input references crashes nix/returns invalid store path #6660

Closed pennae closed 4 months ago

pennae commented 2 years ago

Describe the bug

building a FOD that contains input references either crashes nix or returns a nonexistant store path. crashes happen for FODs with flat hashing, recursively hashed FODs produce an invalid store path.

Steps To Reproduce

  1. nix repl '<nixpkgs>'
  2. :b let a = runCommand "a" { outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "sha256-Q1WkaxnTSNwvV8BG+O9j1FOOu5NgAPPJ7pV1J0YN2GU="; } "echo 1 ${bash} > $out"; in runCommand "b" {} "ls ${a} > $out": daemon crashes with assertion violation

  3. :b let a = runCommand "a" { outputHashMode = "recursive"; outputHashAlgo = "sha256"; outputHash = "sha256-uCxlVcIPQdSr0mbcwNvsfk1ssvrgA/nK2AFNaCM8VjM="; } "echo 1 ${bash} > $out"; in runCommand "b" {} "ls ${a} >$out":

    b> ls: cannot access '/nix/store/pfw3xqs9hz3wx12gbyyc6yl37nychz03-a': No such file or directory
error: builder for '/nix/store/mmqwvwd255apxiy2p34dr7zj7sy5h8m2-b.drv' failed with exit code 2;
       last 1 log lines:
       > ls: cannot access '/nix/store/pfw3xqs9hz3wx12gbyyc6yl37nychz03-a': No such file or directory
       For full logs, run 'nix log /nix/store/mmqwvwd255apxiy2p34dr7zj7sy5h8m2-b.drv'.

Expected behavior

maybe some error message about input references being disallowed in FODs.

nix-env --version output

nix-env (Nix) 2.8.1

Additional context

nix-daemon[2788217]: nix-daemon: src/libstore/store-api.cc:188: nix::StorePath nix::Store::makeFixedOutputPath(nix::FileIngestionMethod, const nix::Hash&, std::string_view, const StorePathSet&, bool) const: Assertion `references.empty()' failed.
systemd[1]: Started Process Core Dump (PID 2788233/UID 0).
systemd-coredump[2788234]: [🡕] Process 2788217 (nix-daemon) of user 0 dumped core.

                               Stack trace of thread 2788217:
                               #0  0x00007f350dce9c1f __pthread_kill_implementation (libc.so.6 + 0x87c1f)
                               #1  0x00007f350dc9f042 raise (libc.so.6 + 0x3d042)
                               #2  0x00007f350dc8a49c abort (libc.so.6 + 0x2849c)
                               #3  0x00007f350dc8a3d5 __assert_fail_base.cold.0 (libc.so.6 + 0x283d5)
                               #4  0x00007f350dc98062 __assert_fail (libc.so.6 + 0x36062)
                               #5  0x00007f350e64ffca _ZNK3nix5Store19makeFixedOutputPathENS_19FileIngestionMethodERKNS_4HashESt17basic_string_viewIcSt11char_traitsIcEERKSt3setINS_9StorePathESt4lessISA_ESaISA_EEb (libnixstore.so + 0x322fca)
                               #6  0x00007f350e4d8e19 _ZZN3nix19LocalDerivationGoal15registerOutputsEvENKUlNS_26DerivationOutputCAFloatingEE7_clES1_ (libnixstore.so + 0x1abe19)
                               #7  0x00007f350e4d9581 _ZZN3nix19LocalDerivationGoal15registerOutputsEvENKUlRKNS_23DerivationOutputCAFixedEE9_clES3_ (libnixstore.so + 0x1ac581)
                               #8  0x00007f350e4d9cb2 _ZNSt8__detail9__variant17__gen_vtable_implINS0_12_Multi_arrayIPFNS0_21__deduce_visit_resultIN3nix13ValidPathInfoEEEONS4_10overloadedIJZNS4_19LocalDerivationGoal15registerOutputsEvEUlRKNS4_30DerivationOutputInputAddressedEE8_ZNS8_15registerOutputsEvEUlRKNS4_23DerivationOutputCAFixedEE9_ZNS8_15registerOutputsEvEUlRKNS4_26DerivationOutputCAFloatingEE10_ZNS8_15registerOutputsEvEUlRKNS4_24DerivationOutputDeferredEE11_ZNS8_15registerOutputsEvEUlRKNS4_22DerivationOutputImpureEE12_EEERKSt7variantIJS9_SD_SH_SL_SP_EEEJEEESt16integer_sequenceImJLm1EEEE14__visit_invokeESU_SY_ (libnixstore.so + 0x1accb2)
                               #9  0x00007f350e4d542e _ZN3nix19LocalDerivationGoal15registerOutputsEv (libnixstore.so + 0x1a842e)
                               #10 0x00007f350e498092 _ZN3nix14DerivationGoal9buildDoneEv (libnixstore.so + 0x16b092)
                               #11 0x00007f350e501bac _ZN3nix6Worker3runERKSt3setISt10shared_ptrINS_4GoalEENS_15CompareGoalPtrsESaIS4_EE (libnixstore.so + 0x1d4bac)
                               #12 0x00007f350e4af640 _ZN3nix5Store10buildPathsERKSt6vectorINS_11DerivedPathESaIS2_EENS_9BuildModeESt10shared_ptrIS0_E (libnixstore.so + 0x182640)
                               #13 0x00007f350e51b316 _ZN3nix6daemon17processConnectionENS_3refINS_5StoreEEERNS_8FdSourceERNS_6FdSinkENS0_11TrustedFlagENS0_13RecursiveFlagESt8functionIFvRS2_EE (libnixstore.so + 0x1ee316)
                               #14 0x0000560dbfe53fcc _ZNSt17_Function_handlerIFvvEZL10daemonLoopvEUlvE_E9_M_invokeERKSt9_Any_data (nix + 0x165fcc)
                               #15 0x00007f350e2deb6d _ZNSt17_Function_handlerIFvvEZN3nix12startProcessESt8functionIS0_ERKNS1_14ProcessOptionsEEUlvE_E9_M_invokeERKSt9_Any_data (libnixutil.so + 0xf0b6d)
                               #16 0x00007f350e2d8b49 _ZN3nixL6doForkEbSt8functionIFvvEE (libnixutil.so + 0xeab49)
                               #17 0x00007f350e2dec29 _ZN3nix12startProcessESt8functionIFvvEERKNS_14ProcessOptionsE (libnixutil.so + 0xf0c29)
                               #18 0x0000560dbfe55287 _ZL9runDaemonb (nix + 0x167287)
                               #19 0x0000560dbfe55a1e _ZL15main_nix_daemoniPPc (nix + 0x167a1e)
                               #20 0x0000560dbfed2a10 _ZN3nix11mainWrappedEiPPc (nix + 0x1e4a10)
                               #21 0x00007f350e84bf59 _ZN3nix16handleExceptionsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt8functionIFvvEE (libnixmain.so + 0x34f59)
                               #22 0x0000560dbfd9b586 main (nix + 0xad586)
                               #23 0x00007f350dc8b237 __libc_start_call_main (libc.so.6 + 0x29237)
                               #24 0x00007f350dc8b2f5 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x292f5)
                               #25 0x0000560dbfd9f3c1 _start (nix + 0xb13c1)
nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fetchzip-results-in-do-not-know-how-to-unpack-source-archive-due-to-wrong-nix-store-path/22014/14

SuperSandro2000 commented 1 year ago

When fetching a patch that contains a nix store path then the build just dies.

3.patch> building '/nix/store/5b7jcfdn1r147bjjvjvp4m6sp6a2f7sl-3.patch.drv'
3.patch>
3.patch> trying https://github.com/serokell/nix-diff/pull/3.patch
3.patch>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
3.patch>                                  Dload  Upload   Total   Spent    Left  Speed
3.patch>   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
3.patch> 100 91015    0 91015    0     0   129k      0 --:--:-- --:--:-- --:--:--  129k
error: unexpected end-of-file
ncfavier commented 1 year ago

There's a duplicate of this at https://github.com/NixOS/nix/issues/7150

roberth commented 1 year ago

In https://github.com/NixOS/nixpkgs/pull/241645 the reference is unintentional. In a fetchClosure call in a source that's fetched by FOD, this may happen too. Both are cases of stray dependencies, and may be valid use cases for unsafeDiscardReferences. Regardless of that, it seems like a good idea for FOD functions to set allowedReferences to [ ], so that the possibility of having fixed outputs with references is controlled by the expressions rather than a fairly arbitrary design or implementation decision.

And of course, the store needs to at least throw a proper error instead of relying on an assertion.

edolstra commented 4 months ago

This appears to be fixed in current Nix (2.23):

nix-repl> :b let a = runCommand "a" { outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "sha256-Q1WkaxnTSNwvV8BG+O9j1FOOu5NgAPPJ7pV1J0YN2GU="; } "echo 1 ${bash} > $out"; in runCommand "b" {} "ls ${a} > $out"
error: fixed output derivation 'a' is not allowed to refer to other store paths.
       You may need to use the 'unsafeDiscardReferences' derivation attribute, see the manual for more details.

nix-repl> :b let a = runCommand "a" { outputHashMode = "recursive"; outputHashAlgo = "sha256"; outputHash = "sha256-uCxlVcIPQdSr0mbcwNvsfk1ssvrgA/nK2AFNaCM8VjM="; } "echo 1 ${bash} > $out"; in runCommand "b" {} "ls ${a} >$out"
error: illegal path references in fixed-output derivation '/nix/store/awnqq6a22sad9p96pcrrhzbyr52b6ibi-a.drv'
error: 1 dependencies of derivation '/nix/store/0a6zvx07dkhzys1ap8pfq5akn9vkrbrl-b.drv' failed to build

nix-repl> :b let a = runCommand "a" { outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "sha256-Q1WkaxnTSNwvV8BG+O9j1FOOu5NgAPPJ7pV1J0YN2GU="; __structuredAttrs = true; unsafeDiscardReferences.out = true; } "echo 1 ${bash} > $out"; in runerror: hash mismatch in fixed-output derivation '/nix/store/wavkn1m3sl1y4lixqc3wnkvmzq65ca81-a.drv':
         specified: sha256-Q1WkaxnTSNwvV8BG+O9j1FOOu5NgAPPJ7pV1J0YN2GU=
            got:    sha256-EFzFdQlZoue6mg6e9jY8WWF1reeI+ZbKOxowFSrc36c=
error: 1 dependencies of derivation '/nix/store/riynv93rm8sjqg8ixy511hln44m8hrsj-b.drv' failed to build