GHC can't find the `unordered-containers` library

[f1rstlady]

Hello, I encountered a strange issue when developing a Haskell project. I used to enter my dev environment with nix-shell, everything worked fine. But nix-direnv seems to mess up the environment such that GHC can't find Haskell's unordered-containers library that was installed by nix.

Minimal working example

I debugged my environment and could reduce the problem to the following minimal working example. A trivial Haskell project set up by cabal:

cabal-version:   3.4
name:            nix-direnv-with-unordered-containers
version:         0.1.0.0

executable test
  main-is: Main.hs
  build-depends:
    , base
    , unordered-containers

And a default.nix file that builds the project through haskellPackages.developPackage, which in turn reads the *.cabal file and installs the specified dependencies:

let
  pkgs = import <nixpkgs> {};
  build = pkgs.haskellPackages.developPackage { root = ./.; };
in build.overrideAttrs (prevAttrs: {
  nativeBuildInputs = (prevAttrs.nativeBuildInputs or [])
    ++ (with pkgs.haskellPackages; [
      cabal-install
    ]);
})

Expected behaviour

The command in question is cabal exec -- ghc --print-libdir. In the original nix-shell, it succeds:

$ nix-shell --run cabal exec -- ghc --print-libdir -Q
/nix/store/x0pn53qh4mnkvfwgm7qm6n19wgk3fyr6-ghc-9.6.5-with-packages/lib/ghc-9.6.5/lib

Actual behaviour

However, in the environment provided by nix-direnv, it fails:

$ cabal exec -- ghc --print-libdir
Warning: The package list for 'hackage.haskell.org' does not exist. Run 'cabal
update' to download it.
Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] trying: nix-direnv-with-unordered-containers-0.1.0.0 (user goal)
[__1] unknown package: unordered-containers (dependency of
nix-direnv-with-unordered-containers)
[__1] fail (backjumping, conflict set: nix-direnv-with-unordered-containers,
unordered-containers)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: nix-direnv-with-unordered-containers,
unordered-containers

Interestingly, if you remove the unordered-containers dependency from the Cabal file, it gives the expected output.

---

I'm aware that this issue is quite specfic since it involves not only nix-direnv but nixpkgs.haskellPackages.developPackage, cabal and the unordered-containers Haskell library. I would be glad if you could give me some hint.

[bbenne10]

I'm unsure of the root cause of this, but I suspect that we'll find differences in the environments created by the two different invocations.

Does nix develop work for you similarly to how the nix-shell invocation does? If not, I think the problem is in how nix develop creates an environment. If so, I suspect we'll see some difference in how the environment variables in between the nix develop and nix-direnv shells. You can see how we handle the environment import here: https://github.com/nix-community/nix-direnv/blob/master/direnvrc#L131. You'll note that we only handle some T{E,}MP*, terminfo, XDG_DATA_DIRS and NIX_BUILD_TOP individually and that we import everything else that's sourced from the output of nix print-dev-env (which is what created the cached $profile_rc on line 149).

I'm afraid I don't have the time immediately to dig into this issue (nor do I have nearly enough haskell knowledge), but hopefully this gives you enough information to go digging. If you have more questions, please don't hesitate to ask. I'm happy to consult as I find time.

[f1rstlady]

Does nix develop work for you similarly to how the nix-shell invocation does?

Hm, I do not use a flake, just a nix-shell setup. Should I create one?

In case it helps, I ran env in both environments and sanitised the output for comparability (replaced newlines in multi-line strings by \\n and sorted the output): nix-direnv.txt nix-shell.txt

[bbenne10]

Without more information about how you're invoking nix-direnv (or a LOT more understanding about Haskell/Cabal), I can't say much more than "yep - those files have different contents".

What is in your .envrc? Is it just use nix?

Here is the spot where use nix is defined. I think you'll find it comprehensible, if not amazingly clear. We simply check if our cached output is out of date (and if we are supposed to be re-generating it based on user settings) and, we have the go ahead, create an array of args to pass to the following nix print-dev-env call on line 471 and then print-dev-env and cache the output.

I suspect that there's some difference in either how you should be calling use nix OR in the environment that's generated. I simply don't have the free time (or Haskell/Cabal know-how) at the moment to go through those environment diffs and suss out what differences may be meaningful.

[paj0sch]

I am experiencing the same issue, using haskellPackages.developPackage, where nix-shell and afterwards cabal run works as expected, with the Haskell libraries provided by nix and available for cabal.

Running cabal run in the direnv environment however, makes cabal fetch/build the packages. I am calling use_nix on a default.nix, so no flake.

One thing I noticed was that my starship prompt calls the direnv environment (vault-0.1.0.0-env) while the one invoked by nix-shell is (ghc-shell-for-vault-0.1.0.0) (where vault is the package name and 0.1.0.0 the package version).

[pranaysashank]

@f1rstlady it's likely the case that cabal & ghc being picked is different in the shell & outside. Try running

nix-shell --run "which ghc; which cabal"

and check outside

which ghc; which cabal

if ghc picked is the same, run ghc-pkg list and see if the unordered-containers package exists in there

@paj0sch this could be related to how nixpkgs' haskell infra works, cabal is free to choose a different plan if it knows there are new packages available in it's index and if the constraints allow it, check ghc-pkg list to see if the package cabal wants to build is available (assuming ghc-pkg & ghc comes from the default.nix in nix-shell). Try,

CABAL_DIR=/tmp cabal run

to not use the available index in your ~/.cabal/store

[paj0sch]

@pranaysashank running your cabal command makes cabal fail in the direnv environment (could not resolve dependencies, unknown package) while succeeding without errors in the nix-shellenvironment (similar to the original problem posted here).

[f1rstlady]

@f1rstlady it's likely the case that cabal & ghc being picked is different in the shell & outside

I don't have ghc and cabal installed in my user environment, this is not the issue.

[pranaysashank]

@paj0sch did you check ghc-pkg list outside the shell to see if it lists the required package?

[paj0sch]

@pranaysashank Yes, the package is not listed in the direnv shell, while being listed in the nix-shell shell. Sorry, that was what I was trying to say in my previous answer.

[pranaysashank]

Edit: See the next comment for the fix for this original issue

@paj0sch I can reproduce the original issue and it seems to be related to how developPackage works. I haven't used developPackage before but with using shellFor as I'm used to I get ghc, cabal in direnv environment as well. If you're curious here's default.nix with shellFor

let
  pkgs = import <nixpkgs> {};

  haskellPackagesWithMine = pkgs.haskellPackages.override {
    overrides = self: super: {
      nix-direnv-with-unorder-containers = self.callCabal2nix "nix-direnv-with-unordered-containers" ./. {};
    };
  };
in haskellPackagesWithMine.shellFor {
  packages = p: [ p.nix-direnv-with-unorder-containers ]; # Add ghc packages you want to develop here
  buildInputs = [ pkgs.cabal-install ] # optional: Add any packages you want in the shell here
}

@bbenne10 do you happen to have any insights into why direnv doesn't get the same environment as nix-shell, here's how developPackage is defined and here's shellFor Edit: Nevermind, I figured it out

[pranaysashank]

@paj0sch @f1rstlady Apparently you need to set returnShellEnv = true; for it to get the shell outside nix-shell as well. So change your default.nix to

-  build = pkgs.haskellPackages.developPackage { root = ./.; };
+  build = pkgs.haskellPackages.developPackage { root = ./.; returnShellEnv = true; };

[paj0sch]

@pranaysashank Oh wow, what an oversight. Thank you very much, the option works as expected, fixing the issue for me.

[f1rstlady]

Thanks, @pranaysashank, for your effort!

Looking at developPackage, it sets returnShellEnv if the IN_NIX_SHELL env var is set. Out of curiosity: Why is it not set in direnv, although it used to be before 3cca1afdec33bc2b860794718c7e2db0c94c168c?

[bbenne10]

@f1rstlady: Are you referring to Line 117 of that diff? If so, we didn't explicitly remove the variable, but rather only stopped shadowing it in nix-direnv. It should still be set to whatever value the underlying invocation sets it to.

I thought that this might be an unnoticed regression in how we invoke non-flake workflows, so I gave it a test but without the GHC toolchain:

In .envrc:

if ! has nix_direnv_version || ! nix_direnv_version 3.0.4; then
  source_url "https://raw.githubusercontent.com/nix-community/nix-direnv/3.0.4/direnvrc" "sha256-DzlYZ33mWF/Gs8DDeyjr8mnVmQGx7ASYqA5WlxwvBG4="
fi

use nix

In shell.nix (or default.nix - I tested both to see if there was a difference based on filename):

let
  nixpkgs = builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/3c80acabe4eef35d8662733c7e058907fa33a65d.tar.gz";
    sha256 = "1q7yfx235bxi3nfg0zm51sjl1akwlilvkhx6p1mf5rwrilb0iln3";
  };
  pkgs = import nixpkgs { config = {}; };
in pkgs.mkShell {
  packages = builtins.attrValues {
    inherit (pkgs) hello;
  };
}

I then simply did echo $IN_NIX_SHELL and got "impure" (for both shell.nix and default.nix). Additionally, +IN_NIX_SHELL is displayed as output from direnv, meaning it is being set:

direnv: loading ~/nix-direnv-hs-test/.envrc                                                                                                                                                                           
direnv: loading https://raw.githubusercontent.com/nix-community/nix-direnv/3.0.4/direnvrc (sha256-DzlYZ33mWF/Gs8DDeyjr8mnVmQGx7ASYqA5WlxwvBG4=)
direnv: using nix
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
direnv: nix-direnv: renewed cache
direnv: export +AR +AS +CC +CONFIG_SHELL +CXX +HOST_PATH +IN_NIX_SHELL +LD +LD_DYLD_PATH +MACOSX_DEPLOYMENT_TARGET +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_TARGET_HOST_x86_64_apple_darwin +NIX_BUILD_CORES +NIX_CC +NIX_CC_WRAPPER_TARGET_HOST_x86_64_apple_darwin +NIX_CFLAGS_COMPILE +NIX_DONT_SET_RPATH +NIX_DONT_SET_RPATH_FOR_BUILD +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_IGNORE_LD_THROUGH_GCC +NIX_LDFLAGS +NIX_NO_SELF_RPATH +NIX_STORE +NM +PATH_LOCALE +RANLIB +SIZE +SOURCE_DATE_EPOCH +STRINGS +STRIP +ZERO_AR_DATE +__darwinAllowLocalNetworking +__impureHostDeps +__propagatedImpureHostDeps +__propagatedSandboxProfile +__sandboxProfile +__structuredAttrs +buildInputs +buildPhase +builder +cmakeFlags +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +dontAddDisableDepTrack +mesonFlags +name +nativeBuildInputs +out +outputs +patches +phases +preferLocalBuild +propagatedBuildInputs +propagatedNativeBuildInputs +shell +shellHook +stdenv +strictDeps +system ~PATH ~XDG_DATA_DIRS
~/nix-direnv-hs-test via ❄️  impure (nix-shell-env) took 2s 

I do not fully understand the implications of these results yet.

[pranaysashank]

@bbenne10 It looks like the variable should be present before the nix print-dev-env command is invoked. The following set of commands with the default.nix in this issue output different envs based on the presence of the env var. Run,

echo "$(nix print-dev-env --profile /tmp/tt --impure --file ./default.nix)" > without-env

echo "$(IN_NIX_SHELL=impure nix print-dev-env --profile /tmp/tt --impure --file ./default.nix)" > with-env

Diffing them both diff without-env with-env shows that it is indeed affecting the resulting env as we get ghc with the correct packages in its environment in the latter case

[bbenne10]

Thanks for tracking this down. I see now how this problem was introduced in the above referenced commit. I am still not entirely sure quite what to do in nix-direnv's code though. I can see that we might desire to explicitly set IN_NIX_SHELL before invoking nix print-dev-env in the use nix case, but I can't guarantee that this has no other negative effects in frankly more common use cases.

Looking over nixpkgs, I can see that there's a number of features gated by pkgs.lib.inNixShell (which is trivially defined to be builtins.getEnv "IN_NIX_SHELL" != ""), so I think this is a good move, but I have a bit more digging to do to see how this works in pure eval mode (which is used for use flake)...

[pranaysashank]

perhaps it should be nix's print-dev-env command's responsibility? (to set the env var)

[bbenne10]

I think that question gets thorny in the nix ecosystem right now. If flakes are the future, you should not be relying on your calling environment at all (for various reasons - some good and some not so good). This means that you should just add returnShellEnv=true; to your developPackage invocation and move on, I think.

If we're supporting "old" workflows (that is - nix-shell rather than nix shell oriented workflows), then the onus of setting that variable is on us, I think, since we are "faking" a nix-shell by invoking print-dev-env under the covers and not exactly advertising that fact to consumers, so patching over the differences falls to us.

I'm going to say - for now - that nix-direnv needs to figure out a way to handle both new/experimental and old/stable workflows equally. (Note that my stance on this in another project would be different. For instance, we don't have this exact problem in flake-env, since there I explicitly don't support this particular code path - there are advantages and disadvantages to "quickly" supporting new workflows :P ).

I will work up a commit that simply sets IN_NIX_SHELL to impure before invoking print-dev-env in use_nix sometime in the near future (unless someone beats me to it). Life's pretty full at the moment, so it may take a few days but I will get to it.

[bbenne10]

I just opened #498 to hopefully address this issue. Would y'all mind testing it out and letting me know if it resolves this particular issue for you?

[f1rstlady]

How am I supposed to test this? I added the following overlay:

final: prev: {
  nix-direnv = prev.nix-direnv.overrideAttrs {
    src = final.fetchFromGitHub {
      owner = "nix-community";
      repo = "nix-direnv";
      rev = "GH-491";
      hash = "sha256-Y3jAE/c7z2cw/YnH6AUmSuEADuJ+oMrNRCLUo/iZrjI=";
    };
  };
}

Afterwards, I rebuilt my home-manager config. This resulted in:

building '/nix/store/flphq755blr6fphdpyqnp1ynpx09c844-nix-direnv-3.0.4.drv'...
Running phase: unpackPhase
unpacking source archive /nix/store/qqz85ya7sv2j2ich0cn81vmwfa2lqvzi-source
source root is source
Running phase: patchPhase
Running phase: updateAutotoolsGnuConfigScriptsPhase
Running phase: installPhase
Running phase: fixupPhase
[resholve context] : invoking resholve with PWD=/nix/store/vmyvkv269b0xr2vx705j8c60d4qcdf87-nix-direnv-3.0.4
[resholve context] RESHOLVE_LORE=/nix/store/d7i149im50d8m7ikvv9wmhqdm29352z5-more-binlore
[resholve context] RESHOLVE_EXECER='cannot:/nix/store/35klgzald67mkslqb9kkv01gn98zfbza-direnv-2.34.0/bin/direnv cannot:/nix/store/j7rp0y3ii1w3dlbflbxlv4g7hbaaz3bs-nix-2.18.2/bin/nix'
[resholve context] RESHOLVE_FAKE='builtin:'\''PATH_add'\'';'\''direnv_layout_dir'\'';'\''has'\'';'\''log_error'\'';'\''log_status'\'';'\''watch_file'\'' function:'\''shasum'\'''
[resholve context] RESHOLVE_INPUTS=/nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin:/nix/store/j7rp0y3ii1w3dlbflbxlv4g7hbaaz3bs-nix-2.18.2/bin
[resholve context] RESHOLVE_INTERPRETER=none
[resholve context] RESHOLVE_KEEP='$cmd $direnv'
[resholve context] /nix/store/lh1njl5y8blxindlx274x3s8w5xhnq9k-resholve-0.10.5/bin/resholve --overwrite share/nix-direnv/direnvrc
Aborting due to missing file: '/nix/store/vmyvkv269b0xr2vx705j8c60d4qcdf87-nix-direnv-3.0.4/share/nix-direnv/direnvrc'
/nix/store/xfhkjnpqjwlf6hlk1ysmq3aaq80f3bjj-stdenv-linux/setup: line 131: pop_var_context: head of shell_variables not a function context

[bbenne10]

In a project specific .envrc, this should work (The git hash and shasum will change if the contents of the PR change, but this is accurate as of now):

source_url "https://raw.githubusercontent.com/nix-community/nix-direnv/54add2fba2e2a0817c9ff1f20750f4ff89a346df/direnvrc" "sha256-Z4PVSVmksQjkRgKfzX49L9TTpG2TZdSjBqtigZt3Y2g="

use nix

This uses direnv's built-in support for fetching URLs and sourcing the result. It'll grab the PR direnvrc and source it and then run the resulting use_nix function (which will override the one in your HM configuration by virtue of being included later).

I'm not sure why your overlay failed, but this avoids doing the resholve build at all and should work fine (If it doesn't, please report back!)

[pranaysashank]

@bbenne10 I tested it and it fixes the issue

[bbenne10]

I just merged #498, resolving this issue. We will have to tag a new release to get it into nixos. I'll have a look at the changeset that we've accrued between 3.0.4 and here and see if we should cut a new release. For now, please use the returnShellEnv workaround or fetch from main for affected project environments

[Mic92]

We can probably make a patch release for 24.05 as well.

[Mic92]

https://github.com/nix-community/nix-direnv/releases/tag/3.0.5

[Mic92]

https://github.com/NixOS/nixpkgs/pull/316897

[bbenne10]

You were faster at doing what I meant to do today - index what we had and cut a point release. Thank you! I don't think backporting for 24.05 will be an issue, but wanted to note that we weren't there yet (probably could have been clearer - hadn't yet had my coffee :coffee:)

[f1rstlady]

After this was fixed, I encountered a different error:

$ cabal build
...
/nix/store/0gi4vbw1qfjncdl95a9ply43ymd6aprm-binutils-2.40/bin/ld.gold: error: cannot open b/outputs/out/lib: No such file or directory
...

where the current working directory is "/home/.../a b", i.e. contains spaces, such that the actual path is split at the space and an invalid path is passed to the linker.

Investigating the diff between nix-shell's and nix-direnv's environments shows that it is caused by NIX_LDFLAGS being set differently:

$ nix-shell --run 'env | rg NIX_LDFLAGS'
NIX_LDFLAGS=-rpath /nix/store/hzqk70r5ixsjsp7j1kn0y25rkxlk7wi9-ghc-shell-for-model-checker-0.1.0.0/lib ...
$ direnv allow
$ env | rg NIX_LDFLAGS
NIX_LDFLAGS=-rpath /home/.../a b/outputs/out/lib ...

Do you have an idea what might have caused this difference?

[pranaysashank]

@f1rstlady spaces in directory are not supported by nix, See

https://github.com/NixOS/nix/issues/842

https://github.com/NixOS/nixpkgs/issues/177952

[f1rstlady]

@pranaysashank Didn't know. Thanks!