`nix profile install` doesn't work with static build of Nix

[ghost]

Describe the bug

If I download the latest static build of Nix from Hydra and try to use its "profile" functionality to install a package (say, nixpkgs#hello, I get the following error:

error: filesystem error: directory iterator cannot open directory: No such file or directory [/nix/store/bw9z0jxp5qcm7jfp4vr6ci9qynjyaaip-hello-2.12.1].

The binary apparently gets installed, but running it fails with the following error:

bash: /home/user_libvirt_fedora40/.local/share/nix/root/nix/store/bw9z0jxp5qcm7jfp4vr6ci9qynjyaaip-hello-2.12.1/bin/hello: cannot execute: required file not found

Steps To Reproduce

1. Go to https://hydra.nixos.org
2. Navigate: nix -> master -> Jobs -> buildStatic.x86_64-linux
3. Go to build 259117107
4. Download nix to ~/.local/bin
5. Make it executable: chmod +x ~/.local/bin/nix
6. Run nix --extra-experimental-features nix-command --extra-experimental-features flakes profile install nixpkgs#hello
7. Observe that a download is taking place, but in the end the following error is displayed: error: filesystem error: directory iterator cannot open directory: No such file or directory [/nix/store/bw9z0jxp5qcm7jfp4vr6ci9qynjyaaip-hello-2.12.1]
8. Observe that the following file exists: ~/.local/share/nix/root/nix/store/bw9z0jxp5qcm7jfp4vr6ci9qynjyaaip-hello-2.12.1/bin/hello, but attempting to execute it results in the following error: bash: /home/user_libvirt_fedora40/.local/share/nix/root/nix/store/bw9z0jxp5qcm7jfp4vr6ci9qynjyaaip-hello-2.12.1/bin/hello: cannot execute: required file not found
9. Observe that ~/.nix-profile is a broken symlink to ~/.local/state/nix/profiles/profile, as the latter path doesn't exist. Creating this directory doesn't change the outputs of the commands mentioned above.
10. Observe that nix --extra-experimental-features nix-command --extra-experimental-features flakes run nixpkgs#hello outputs Hello, world!, as expected

Expected behavior

Installation should have been completed successfully, the binary should run

nix-env --version output

Apparently, not applicable with a static build...?

Additional context

I would like to describe the intended use case here.

My goal is to use a static build of Nix in the same manner one would normally use conda/miniconda/miniforge on an HPC cluster. In this environment I don't have root privileges, so I can't install Nix "the normal way".

As I understand there are ways to get it running using various workarounds from third-parties, but if at all possible, I would prefer to go with official builds.

So ideally I would like to be able to download some standalone binary from Hydra, do something like nix ... install my long list of packages and have the corresponding executables runnable from my shell whenever I log in.

Please let me know if I am approaching the whole problem from the wrong angle. It is very likely that I got something wrong about how I should go about installing a static build, and probably I should have configured it in some manner, but the fact that the commands I mentioned above do create a user-local nix store but later fail to resolve what looks like a system-global nix path makes me think that there is also a bug somewhere.

I would be glad to help debug this further if it is not a misconfiguration on my part.

[fricklerhandwerk]

Triaged in Nix team meeting:

@serprca this is exactly the intended use case for the static binary, and we definitely want it to work.

• @edolstra: could be a std::filesystem regression (a recent change)
• Assigned to @Ericson2314, this is probably something about nix profile not being able to run binaries in chroot
• @edolstra: nix run already runs the executable in a chroot that matches the store it lives in

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-05-15-nix-team-meeting-minutes-146/45491/1

[CallumMcMahon]

Just commenting that I'm in a similar situation on a HPC, and having working non-root stores working smoothly would be amazing for replacing conda workloads. I currently have a working setup using nix-user-chroot but needing to enter a sub-shell every time is a bit finicky. I stumbled across a similar report on the forums which could make people think it's intended behaviour. Also saw a recursive nix method mentioned here which I think is also to get around rootless nix profile limitations, but think that may have broke my nix-profile and generally seemed like unnecessary complexity.