"Lighter" and "cleaner" shell for `nix develop` and `nix shell`

garbas commented 3 years ago

Proposal

Final goal is to provide better integration with other shell environments - eg. zsh, fish, ...

Depending how this will be implemented a very nice to have would be to not spawn a new process when entering shell environment. This would mean the current user shell environment would be updated in-place (like direnv does it).

Reasoning

For those who don't use bash using nix-shell, nix develop or nix shell usually means you need to run nix-shell --run $SHELL or creating an alias.

I understand there are good reasons for using bash. Using bash ensures that development environments are very predictably and running commands from nix-shell would act as close as it can within a sandbox.

But on the other side we are forcing developers to use bash or use workarounds. As all debates around editors also debates around shell environments are a hot topic among developers (which is our audience in this case). I wouldn't expect developers switching to bash just so they can use Nix. I would actually expect the opposite, potential users being turned away from Nix due to the lack of good integration with their shell environment.

As with most of the UX decisions also this comes down to compromising one thing over the other.

Bonus

Another bonus to consider is also which environment variables we should expose inside the shell environments.

From the perspective of a developer who just wants to enter a development environments to be able to develop their project, all the extra variables that get created confuse. This "type" of developers are not interested in building a project with Nix, but rather only interested in using Nix to provide needed development environment.

From the perspective of a developer who is working on a tool that will be built later with nix build they want to see - especially when packaging -

We already have options to ignore environment variables (via --ignore-environment flag). We can make this a default and maybe even improve further and provide even "cleaner" development environments.

Ericson2314 commented 3 years ago

nix shell and nix develop are geared for debugging Nix builds. We should have a different command for just using things that doesn't even bother with stdenv / genericBuild (which use/do bashism and are slow).

abathur commented 3 years ago

I was initially going to post in agreement with @Ericson2314, but after reviewing the docs I'm unsure. Clearing up messaging and managing expectations seem like preconditions for thinking this through. I have tried to keep my head in the sand on both flakes and new-style commands, so maybe I'm a good rubber duck here...

nix --help says (with irrelevant items removed):

Main commands:
  build    build a derivation or fetch a store path
  develop  run a bash shell that provides the build environment of a derivation
  repl     start an interactive environment for evaluating Nix expressions
  run      run a Nix application
  shell    run a shell in which the specified packages are available

I read this as:

nix develop (iirc briefly nix dev-shell) seems to be intended to meet the primary ~package-development need nix-shell was built for.
- It seems like it's best if this continues to use bash for similarity with the build environment.
nix run seems to be intended to meet the "I don't want this in my profile but I want to use/try it real quick" needs that have been using nix-shell -p blah --run/--command. (aside: the short help text for run is probably not enough for a new user to figure this out)
nix shell seems to be intended to take on all of the "nix-managed development environment" needs that have attached to nix-shell over time.
- It seems like this could use any shell and should make it easy to do so (and shell.md suggests it may already use $SHELL) https://github.com/NixOS/nix/blob/199081ad00e6ee4c704eaac34211b454fe0f310c/src/nix/shell.md#L45-L46
- This conflicts with what @Ericson2314 said about nix shell being geared for debugging builds, so I may be misunderstanding. It may also be a sign that it's hard enough to think clearly about what nix shell does given our history with nix-shell that it would be better to rename this and have no nix shell.
The above make me think the goal is to eventually deprecate and remove nix-shell, but I don't know if that's true.

FRidh commented 3 years ago

This conflicts with what @Ericson2314 said about nix shell being geared for debugging builds, so I may be misunderstanding. It may also be a sign that it's hard enough to think clearly about what nix shell does given our history with nix-shell that it would be better to rename this and have no nix shell.

@abathur yes, nix shell (without the dash) is not for providing a build environment.

For that reason, what @garbas suggests should not apply for nix-shell and nix develop, but could for the others.

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/always-run-my-shell-in-nix-shell-nix-develop-et-al/12077/3

blaggacao commented 3 years ago

So that we +- get an idea who we're talking about:

The "devshell persona" Jeff, age 37, living with his young family in Addis Ababa, Ethiopia, is a guy who is responsible for the productivity of a global team of developers, he is driven by the following:

ensure fellow collegues get up to speed with their developer environment in as less time as possible (r13y + cache + CA)
implement so called "shift-left" strategies. A good example is flakes-utils-plus, which runs it's CI through devshell, but also devos which runs a git pre-commit through devshell.
index the developer entrypoint (e.g. MOTD)
elegantly compose domain environments over a monorepo or across organizations (module system + evtl. ad-hoc concatenation of multiple envs in PATH)
avoid "gut rejection" of an unknown cryptic language (nix), use toml files, hide nix where possible
[ please feel free to complete ]

DavHau commented 3 years ago

I guess in general there are two major use cases:

debug a build.
create a dev environment with tools suitable for developing on a project.

For nr. 1 (debug build), it's very simple. One wants to have an environment that is exactly the build environment with all variables and the exact same shell (bash). (this is currently what nix develop does given any derivation)

For nr. 2 (dev environment) it is more difficult to define. I guess we all agree that a dev environment shares things with the build environment. But how to we want to define a dev environment?

should it be based on the build environment with all its vriables + extended with stuff?
should it be defined from scratch, manually specifying all required packages/variables

Maybe I'm biased because of frameworks/tools I like to use, but in my opinion a dev environment should not be based on the build environment. It is usually a separate environment with significantly different properties compared to the build environment.

If we agree on this, then I guess we might be better of renaming nix develop to nix debug. It is not useful for developing usually. According to the manual nix develop provides the build environment of a derivation. This is useful for debugging builds. Not for developing.

Also nix develop/nix debug defaulting to load the devShell output might be wrong. Why would one ever want to debug their devShell? Normal derivations can require debugging, but not a devShell.

Wouldn't it be better if the devShell was meant to be loaded by nix shell?

Thra11 commented 3 years ago

Recently, I've been (ab)using nix develop with shell.nix based on this example in the wiki. By the sounds of it, this isn't the "blessed" way to do general (as opposed to nix) development, but the alternative using nix shell with a defaultPackage is only a TODO at the moment, so I don't know how to do that.

From my point of view, this is what I think I need in a nix shell:

For nix package development (I've never really used this sort of thing much. Trying build and looking at the output has generally been adequate for my basic nix packaging needs):

Needs the relevant hooks.
Should be as pure / close to the package build as possible.
Might need a few extra tools to study the state before or after running various hooks, but ideally nothing that could be detected by the build tools.

For general development:

Doesn't need nix build hooks.
Probably doesn't need sources for the package we're developing, as we'll be working on a VCS-controlled working copy.
Needs:
- Development build inputs such as headers, libs, often a compiler-with-package-set sort of thing.
- Development tools: not just the minimal tools required to build a package, but also repls, IDEs, version control, a user-friendly shell such as zsh, and so on
- Some environment variables from the system environment. For example GUI tools may need environment variables to be set in order to pick up the system GUI configuration and interact with system services.

At this stage, I'm honestly not sure whether this general development use case is supposed to be supported by nix shell or not. If it wasn't for this bug report, I probably wouldn't have realised that I'm not supposed to use nix develop --command zsh for this.

drupol commented 2 years ago

I'm switching my dev environments from regular shell.nix files to flakes.

I now use nix develop instead of nix-shell. The thing is that nix develop starts a bash shell instead of my current shell (fish).

In order to continue using fish, I need to do: nix develop --command fish and everything is like before!

Is there another way and/or another recommended way to do that?

mroi commented 2 years ago

I put a shellHook in my development flakes, where I exec the shell. (Or to be precise: I source a global script, which does that.)

matklad commented 2 years ago

If we agree on this, then I guess we might be better of renaming nix develop to nix debug. It is not useful for developing usually.

+1 to de-confusing "I want to debug a nix-based builds" and "I want to use nix to create a dev environment for hacking on a random project (which might not be even be built with nix)" use cases. My prefered color for the bikeshed would be nix build --interactive for what nix develop currently does.

nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-nix-develop-supposed-to-be-used-for-creating-development-environments/17687/1

roberth commented 2 years ago

Before adding a nix command that is inevitably going to be a bad fit again, consider writing a shell runner derivation that's usable with nix run. By expressing the shell logic in expressions, it is easier to improve and even replace by the community. The nix package itself should remain as simple as possible, because its logic can not be pinned. Perhaps an "alias" for nix run with a different default attribute would be convenient, but that's about as far as we should go, until proven otherwise.

nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-nix-develop-supposed-to-be-used-for-creating-development-environments/17687/6

nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/shell-nix-but-with-flakes/18775/2

williamvds commented 1 year ago

To add to @Thra11's use-case:

Needs:

* Development build inputs such as headers, libs, often a compiler-with-package-set sort of thing.

* Development tools: not just the minimal tools required to build a package, but also repls, IDEs, version control, a user-friendly shell such as zsh, and so on

* Some environment variables from the system environment. For example GUI tools may need environment variables to be set in order to pick up the system GUI configuration and interact with system services.

As a C++ developer, my dev shell is typically a superset of my package's inputs, effectively concatenating buildInputs ++ propagatedBuildInputs ++ nativeBuildInputs ++ ....

That gives me all I need to build, test, and run my software in a close reproduction of a nix build environment. I add extra inputs for additional development tools that every developer would use during development.

I also need to disable (some) hardening, so I can build and debug without optimisations. With mkShell, I add hardeningDisable = [ "fortify" ]; } to do this. (This bit took me a while to figure out. NIX_DEBUG=1 helped me realise Nix's GCC is wrapped in a script which implicitly appends some hardening flags, some of which enable optimisation. I'll look to update some documentation)

Edit: I discovered that the inputsFrom attribute of mkShell does something very similar to what I was looking for, with taking inputs from a derivation (but for some reason checkInputs is omitted). The hardeningDisable override is still necessary however.

jakekreider commented 1 year ago

Being new to Nix and Flakes, I was confused by the distinction between nix-shell and nix shell. I have a local workspace/environment with some specific binary versions I want to reference for only this project. With nix-shell and a default.nix file, I was able to decompose it into multiple files, and do myLocalPackage = import ./somefile.nix {} to clean it up.

But when I try the same syntax in a flake.nix and running with nix shell or nix develop, it tries to import the file relative to the nix store, and throws error: getting status of '/nix/store/skk5b55sdlfmh7z68cmlfz2ff9p9n8gd-source/somefile.nix': No such file or directory.

Apologies if this is a totally different issue, but I suspect it's related to misunderstanding of using flakes for building packages vs. for a local development environment.

roberth commented 1 year ago

I was confused by the distinction between nix-shell and nix shell.

This is a known issue that we should resolve. An extensive discussion has occurred in #4715, so "all that remains" is to make a decision.

But when I try the same syntax in a flake.nix and running with nix shell or nix develop,

You probably want nix develop, unless you want to start a relatively clean shell with a locally defined package. These commands will only load a flake.nix and anything imported from there.

it tries to import the file relative to the nix store, and throws error: getting status of '/nix/store/skk5b55sdlfmh7z68cmlfz2ff9p9n8gd-source/somefile.nix': No such file or directory.

Flakes are (currently) copied to the store before evaluation. If a file is missing, often you'd have to git add (or git add -N) it. With #6530 we have an opportunity to improve the error message by checking for an untracked file before throwing the error.

jakekreider commented 1 year ago

Flakes are (currently) copied to the store before evaluation. If a file is missing, often you'd have to git add (or git add -N) it.

That was exactly my problem - I didn't realize the connection between the Nix store and the source tree, even for local dev. Thanks for your help!

nixos-discourse commented 1 year ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/defining-multiple-nix-shell-environments/25770/1

nixos-discourse commented 1 year ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-shell-nix-shell-and-nix-develop/25964/11

mrjones2014 commented 1 year ago

For those finding this issue via a search engine, if you're just looking to have nix-shell run a Fish shell instead of bash, here's what I did to make it work:

Add a file at ~/.config/fish/functions/nix-shell.fish -- this function will wrap nix-shell:

function nix-shell --wraps nix-shell
    # if `--run` was passed at the command line, run as-is
    for ARG in $argv
        if [ "$ARG" = --run ]
            command nix-shell $argv
            return $status
        end
    end

    # otherwise, append `--run "exec fish"` to the argument list to get a Fish shell
    command nix-shell $argv --run "exec fish"
end

mrjones2014 commented 1 year ago

Doesn’t work for nix develop though unfortunately…

ilyagr commented 1 year ago

https://github.com/MercuryTechnologies/nix-your-shell makes nix develop work with fish (or zsh). I found I had to set my prompt manually, see the wiki for a hint of how to do it.

ilyagr commented 1 year ago

I did run into https://github.com/NixOS/nix/issues/6982, which is strictly-speaking related to bash, but did not feel light or clean. It very much affects running bash from a nix develop fish shell (if I wanted to run a one-off bash command).

See https://github.com/NixOS/nix/issues/6982#issuecomment-1637001855 for my ugly workaround.

tobiasBora commented 1 year ago

Since nix develop -c zsh already provides all environment variables, the only missing part is a way to run the build phases automatically. A trivial possibility is to create a mini bash script that runs the various phases, that you can call from any shell. I did a proof of concept there and it worked greatly, and is shell agnostic : https://discourse.nixos.org/t/making-a-development-flake-for-bespoke-synth/29988/5 One could also add in the script a way to save and load the environment/path of the previous command, saved into a file, but this should be quite trivial to do using env.

roberth commented 1 year ago

quite trivial to do using env.

Variables can be local, they can be arrays, and they can be functions. This would need something like declare -p if not more, and you'll probably have to deal with some idiosyncrasies along the way.

I doubt that a true solution for shell interoperability should be developed as part of Nix.

cumber commented 1 year ago

Since nix develop -c zsh already provides all environment variables, the only missing part is a way to run the build phases automatically.

To be honest, I want the opposite. I want the ability to get into a Zsh (or even Bash for that matter) session with just things like $PATH settings added to my normal interactive shell profile, and keep the build phase stuff completely out of it. "I want to be in the nix build environment to run nix build phases" is just not the same concept, to my mind, as "I want to be in a shell environment with a set of packages available".

If I'm actually running Nix build phases, I'm happy to do so from Bash. It's a Bash environment, and not everything you can do in Bash crosses process boundaries into other shell environments nicely. I don't know whether Nix actually uses any features that wouldn't translate well, and I don't want to have to know that in order to predict the behaviour of the resulting shell environment. I would also much prefer not having to know that there's a big list of environment variables (including generic-sounding names like name, out, outputs, shell) that I should expect, ignore, and avoid using similar names if I want to avoid clashes.

Aliases for nix <shell or develop> "$@" -c <shell-of-choice> give me most of what I want in practice, because the stuff that nix defines doesn't tend to actually get in the way that much. But fundamentally it's a shell launched inside an environment that is a tangled mix of the environment variables for dependencies and ones for the nix build framework. If development effort were to be spent on this, I'd much rather it went into untangling the two so that it's possible to get the dependency environment variables without the build framework ones, rather than trying to make the build framework work in multiple shell languages.

neirenoir commented 1 year ago

To be honest, I want the opposite. I want the ability to get into a Zsh (or even Bash for that matter) session with just things like $PATH settings added to my normal interactive shell profile, and keep the build phase stuff completely out of it. "I want to be in the nix build environment to run nix build phases" is just not the same concept, to my mind, as "I want to be in a shell environment with a set of packages available".

Adding onto this: it may not be a bad idea to get inspired by Microsoft's DevContainers. Basically, what I want is to run an isolated, portable development environment I can configure to my liking, without necessarily having to think about packaging it for Nix. I'm looking into build-vm to see if it more or less does what I want it to do, but it's still hacky; I also looked into developing a bwrap-based sandbox for flakes, but shellHook is too clunky for this.

Some setting that allowed me to override the command executed (instead of bash) would be more than enough. Something like what devenv is doing, but not as overkill.

domenkozar commented 8 months ago

In devenv we're exploring this: https://github.com/cachix/devenv/issues/330#issuecomment-1975018698

NixOS / nix