Why is there no way to run `nix-shell` in a chroot and without the user's .bashrc?

[rrnewton]

I'm using the Haskell stack tool's nix integration, which launches everything through a nix-shell.

Even running with --pure, nix-shell seems really impure compared to nix-build. It not only mounts directories, it sources the bashrc from the host system!

Is there any way to lock down nix-shell more using current configuration options? If not, is there any plan to make nix-shell more pure?

This is especially concerning because I thought that shebang lines with nix-shell were a great way to get reproducible scripts. But now I realize that those scripts are much more impure and less reproducible than I thought.

[domenkozar]

Relevant: https://github.com/NixOS/nix/issues/625

[zimbatm]

I submitted a fix a long time ago but it never got merged: https://github.com/NixOS/nix/pull/605

[zimbatm]

Other related:

• https://github.com/NixOS/nix/issues/787
• https://github.com/NixOS/nix/issues/730

[rrnewton]

Great that there's already a fix in #605! If there's any way to draw attention to that I'd like to know whether there's any objection to it, or the PR just lost in the mix.

Another context we have where we really want a pure nix-shell is benchmarking. We run benchmarks through nix to make sure their dependencies install reliably. But if we're running the benchmark scripts themselves through nix-shell, then that compromises their reproducibility.

The only interim workaround I can think of is to make a default.nix for each script we want to run which makes it into a derivation that we can run with nix-build. But that sure pollutes our store with a lot of garbage. Still, if we can copy out what we need and GC the garbage maybe it's an ok solution for now.

[zimbatm]

I don't know if nix-shell is the right solution. It also suffers from other issues like changing HOME or SSL_CERT_PATH depending on the dependency set. These things only make sense in a sandboxed build context.

What I do now is build custom profiles for each of my projects and then change the PATH to the profile-dir/bin (plus a couple other common ones). I use direnv to automate this but you could also just write a wrapper script. nix-env -p $PWD/nix-profile -f . -ir; export PATH=$PWD/nix-profile/bin

[rrnewton]

Ah, but when switching with direnv you don't actually totally clear the environment or chroot?

What do you recommend a tool like the Haskell stack tool do? It currently generates nix-shell commands. I guess the alternative is that it could run a nix-build, but it needs some way to include selective impurities wherein it can "mount" ~/.stack/ and pwd so that it can have some side effects on the file system.

[zimbatm]

I use direnv to add more stuff to an existing shell. Like more dev dependencies for a project. It allows me to jump around projects in the shell.

If you want a new "pure" environment I would recommend using store_path=$(nix-build . --no-out-link), keep all the pure stuff in the build and then use that result for further work. I think that for stack, the ~/.stack folder is just used for build settings and would be used to generate a default.nix file.

[rrnewton]

Well the ~/.stack directory also stores build products, as does ./.stack-work. The philosophy with the stack/nix integration is to let stack handle all the Haskell compilation and use nix for non-haskell dependencies.

Stack's current approach of dropping into nix-shell (and calling itself inside the nix-shell) helps it access all the nix-built dependencies, but it doesn't do much for purity.

@YPares didn't think nix-build should be used to launch stack builds. I don't totally follow that reasoning given that the build output could be trivial or empty in this case and could get garbage collected anyway.

I still think that nix-build rather than nix-shell could at least be an alternate mode that I could potentially add to stack -- as long as there is some way of "mounting" specific additional directories into the build environment for a nix-build. But I don't currently know a method for that.

To bring it back to this issue -- if nix-build can add a bit of impurity with mounts, then it removes the need for a more pure nix-shell.

[zimbatm]

It's the same really, you can use nix-build to compile a buildEnv that includes the ghc and dependency libraries that you want. It's even better because you don't have to do that re-invocation trickery, just change stack's PATH to the derivation result/bin. Maybe also set whatever env GHC uses to find libraries, and then invoke ghc to compile the directory dependencies, which you can still store in ~/.stack.

[rrnewton]

Ok, I think I follow that. But is the approach of using a buildEnv any more pure than using nix-shell like stack is doing now?

In extreme cases it seems like IO that happens during the compile (like via TemplateHaskell) could touch arbitrary impure state.

But maybe we should just do some sandboxing of the call to nix-shell itself. Like changing HOME and clearing env, and maybe chroot'ing ourselves.

[zimbatm]

nix-build is sandboxed if the user enables nix.useChroot = true; so all your project dependencies and ghc are built in that pure environment. stack already installs ghc inside of ~/.stack when asked to do so, it could do the same but instead put a symlink to the ghc+dependencies derivation.

[deepfire]

+1

[Ericson2314]

Wish granted! https://github.com/NixOS/nix/blob/master/src/nix/run.cc

[zimbatm]

It doesn't look like bash is being run with the --noprofile and --norc flags : https://github.com/NixOS/nix/blob/master/src/nix/run.cc#L121

[Craig-Macomber]

I believe this same issue is what causes my Linux Mint 17.2's /etc/bash.bashrc to break nix-shell: $ nix-shell --pure -p hello dircolors: no SHELL environment variable, and no shell type option given The resulting shell fails to get its prompt and PATH set correctly (looks just like the shell you started in) resulting in confusion for new users (such as myself) before making it past the quick start guide. Removing the /etc/bash.bashrc file fixes this, but isn't a great solution.

[deepfire]

I can confirm that /etc/bash.bashrc is being sourced by bash run by nix-shell on Mint 17.2 -- essentially breaking everything, through PATH being overridden.

[bjornfor]

@domenkozar: You labelled this as "feature", but isn't it more "bug"? (Why is "nix-shell --pure ..." sourcing host bashrc files?) Bumping https://github.com/NixOS/nix/pull/605.

[reckbo]

Could we not just build a Bash that doesn't source a global profile by default? i.e. get rid of this

NIX_CFLAGS_COMPILE = ''
-DSYS_BASHRC="/etc/bashrc"
...

[CMCDragonkai]

Since https://github.com/NixOS/nix/issues/1147 was closed as a duplicate of this, I was wondering to what extent is the sandboxing features discussed in https://github.com/NixOS/nix/issues/179 going to be available to nix-shell? Specifically with regards to network namespaces, where sometimes you may be developing a networked application that binds to ports, which means without a network namespace, the application takes over a global resource (the port number). I've been using unix domain sockets where possible, but not all applications support using unix domain sockets as alternatives.

Also what about nixos-containers, where does that fit in between nix-shell and nix-build?

[bryaan]

@CMCDragonkai This definitely brings up an interesting debate. Sandboxing nix-shell by using a chroot absolutely makes sense. It also makes sense one would want to take that to the next step and isolate networking as well. Now I'm pretty sure those 2 features, and everything else available under nix-shell, are the same thing as nixos-containers.

If that is actually the case then I think these tools need to be merged.

[deepfire]

Just one minor thing -- since nix-shell is often used as an interactive environment for fooling around (aka "just give me my deps") -- the disabling of networking should be optional (even if default). I mention this only since I'm unaware of the extent that this non-isolation of networking is provided by nixos-containers.

[deepfire]

Come to think of it.. having an option of keeping unrestricted filesystem access is a useful feature as well.

This allows you to keep custom cross-component ties for the dirty experimentation phase, when you develop your piece of software.

The chroot-style mandatory purity can actually be counter-productive for such exploratory scenarios, since it raises the friction for makeshift composition (or decomposition) considerably. For example when you have an on-going experiment with splitting your package in two, you might want to symlink stuff around.

So I guess it's a no-no to let the shell environment's dependencies (especially such as bash) break that purity on their own, whereas it should be optionally allowed for whatever happens inside the shell, once it's entered?

[CMCDragonkai]

It should be compositional. You start with the minimum-level of isolation, and you should be able to "functionally" compose extra levels of isolation and mix-and-match what you want. This would take a lot of steam away from docker like tools, since they are often heavy weight all or nothing things. At the beginning maybe I just want certain packages available, then maybe I want a custom FS, then maybe network namespaces, then maybe PID namespace, then maybe resource isolation.... etc. Making these things compose will preserve the simplicity and exploratory nature of nix-shell without making it too complex immediately.

[zimbatm]

One difficulty is that nix-shell mixes two different scenarios together: debugging a build and creating a dynamic profile. These two don't have exactly the same needs and nix-shell isn't really good at neither of those so it should be deprecated.

In the debugging scenario, a build has failed and the developer wants to figure out why. He could edit the nix code and re-run the build but that isn't necessarily fast. In that scenario I believe that a --interactive option to nix build where the user is dropped into the sandbox shell would be much better. nix-shell doesn't reproduce the build environment faithfully enough.

In the dynamic profile scenario, the user just wants a bunch of tools and environment variables setup to work on a source tree. Ideally that environment can be pinned to avoid uncontrolled rebuilds when the channel is updated. Even better would be to have an activation script to start project-related services like postgresql and redis. nix run is a bit closer to that scenario because it doesn't pollute the environment with build phases and other build-specific things like nix-shell does. When run with no argument it builds and loads the default.nix.

[bryaan]

@CMCDragonkai I think nix is already well ahead of docker in terms of philosophy, it's just that the world has yet to catch up, and I believe nix need more marketing to really push the testing and eventual adoption. Now bring in orchestration with Hashicorp's consul and nomad and we really have something here. The unix philosophy of tooling is winning here. Docker got it wrong.

@zimbatm Yes, something like deprecating nix-shell and duplicating any feature it currently possesses to nixos-container, such as ability to run on non-nix host natively, or have a solid replacement feature which is just as functional.

[thedavidmeister]

are there clear step by step instructions somewhere for a nix noob like myself to get into a shell that has my deps from shell.nix in it without sourcing global things from the user?

this is hurting adoption in my team as people already have rust installed under their user, shell.nix installs a fresh cargo with the mozilla overlay rather than rustup but the shell shows the existing rust version with cargo -v. Even with nix-shell --pure we see the wrong rust versions coming through, which among other things means that our fmt produces inconsistent formatting, effectively putting nix in the "too hard" basket mentally for people who are coming at this from an ubuntu/mac setup. I'm seeing solutions like globally reinstalling rust through Makefile tasks popping up, even though we have a shell.nix file :(

i'm sure there is some solution (like nix-build with buildEnv which was mentioned above in the comments) but it wasn't obvious to me how to migrate from nix-shell to nix-build because one gives an interactive shell and the other outputs a file... could someone point me towards a ELI5 wiki article or similar that i can work through to get a properly isolated shell?

[7c6f434c]

@thedavidmeister do you have user namespaces? Also, where is your Nixpkgs checkout located? I could try writing an nsjail command for running nix-shell in a much-pruned environment (I use nsjail to sandbox Firefox so I had to learn most of the flags)…

[thedavidmeister]

@7c6f434c i don't know?

my situation is that i've asked my team to install nix on their existing machines (ubuntu and mac) with:

curl https://nixos.org/nix/install | sh

as per https://nixos.org/nix/download.html

then i ask them to run nix-shell --pure in this repo:

https://github.com/holochain/holochain-rust

and then run a command like cargo -v to see the version of cargo used.

if the team member has already used rustup to install cargo then they typically see that version of cargo rather than the one pulled in by the moz overlay from shell.nix

what i'm expecting/hoping is that i can ask a team member to install nix-shell and run the .nix file from the repo and they see only the nightly-2018-10-12 version of cargo that i specified and not whatever is coming in from .bashrc etc.

[7c6f434c]

@7c6f434c i don't know?

my situation is that i've asked my team to install nix on their existing machines (ubuntu and mac) with:

Oh, macs… if you need to cover macOS, nsjail should not be suitable.

[thedavidmeister]

@7c6f434c well, is there at least an ubuntu solution? i'm interested in this nsjail idea

[7c6f434c]

This needs a Nixpkgs checkout made with git clone, and only passes through ~/tmp, lightly tested:

( NIXPKGS=/home/repos/nixpkgs ; nsjail -R /nix -T /tmp -R $NIXPKGS -B /dev -E NIX_PATH=$(dirname "$NIXPKGS") -E "HOME" -T "$HOME" -B "$HOME/tmp" --disable_clone_newnet --rlimit_as max --rlimit_cpu max --rlimit_fsize max --rlimit_nofile max --rlimit_nproc max --rlimit_stack max --skip_setsid -Q -D "$HOME" -- $(nix-build --no-out-link  "$NIXPKGS" -A nix)/bin/nix-shell --pure -p hello )

[dicarlo2]

Can we add an option to disable the source ~/.bashrc; call here: https://github.com/NixOS/nix/blob/master/src/nix-build/nix-build.cc#L411 ? I might be missing something, but it seems like this would cover most of the use case (e.g. this would solve @thedavidmeister's issue).

[dicarlo2]

Also, FWIW, setting PS1 to an empty string and running nix-shell --pure --keep PS1 also achieves the desired effect.

[stefandeml]

nix-shell --pure --keep PS1 still sources .bashrc afaik

edit: looks like there is a fix coming for this: https://github.com/NixOS/nix/pull/3131

[stale[bot]]

I marked this as stale due to inactivity. → More info

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/building-nixpkgs-from-scratch-on-non-nixos/16962/1

[stale[bot]]

I marked this as stale due to inactivity. → More info

[DavidDTA]

This is still a desired feature.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/trying-to-use-nix-in-ci-java-io-ioexception-unknown-host-github-com/20660/3

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/speeding-up-similar-builds-with-oci-layers/45285/6