Nix integration is not supported in new-build.

[ElvishJerricco]

At least, the cabal-install one can compile with the current latest release tag (with GHC 8.2.1) does not support it.

[ElvishJerricco]

It seems --config-file also does not work in new-build. Is there common global command line option logic not being used?

[23Skidoo]

/cc @ttuegel

[ttuegel]

I intentionally did not implement Nix integration for the new-build command because it was yet somewhat unstable. Nix integration will be of limited utility with new-build, although it will be useful to manage foreign dependencies. Unfortunately, I do not expect I will have time to implement this feature soon.

[hvr]

Re --config-file, see #4649

[ElvishJerricco]

@ttuegel What are the challenges involved in implementing this? Do you think it's a reasonable goal for someone like me who has never contributed to Cabal before?

[dcoutts]

What is the goal exactly for nix integration with new-build?

To translate into nix exprs? To build C deps using nix? To do the whole build using the cabal CLI but use nix under the hood to do the builds?

[ElvishJerricco]

In a cabal2nix issue, I briefly outlined a common workflow with Nix+Cabal that I use for single-package projects quite often. The goal (to me) would be to have Nix provide you with the GHC version and all your haskell dependencies, while allowing Cabal to do the incremental development work. The main reason is that Nix builds are much more suited for production than they are for development, while Nix shells are fantastic for development. So cabal new-build --enable-nix would enter a nix-shell to find the GHC it needs, which has a package database with all the necessary packages already present, and it would use that GHC to give you incremental compilation across packages in your project.

[dcoutts]

So having all of a project's non-local dependencies installed via nix means translating the project plan into nix exprs and using nix to execute that. It's hardly any more work to also be able to make nix exprs for building the local deps too, which would be super-useful for CI.

BTW, I don't think single-package nix exprs really make sense for us, a project corresponds to a particular set of packages with particular configuration for them all. Or to put it another way, cabal new-build's nix integration / cabal2nix should really be whole-project not single package.

[ElvishJerricco]

Not quite. The purpose of cabal2nix is to generate a Nix expression that just trusts whatever your nixpkgs.haskellPackages provides as its package set. This workflow does not use Cabal's dependency planning system whatsoever. Nixpkgs acts a bit like Stackage in this regard, and Cabal just becomes a great command line interface for incremental development over that.

[dcoutts]

Ok, whereas for cabal new-build, it'd ignore nixpkgs.haskellPackages and do exactly what your cabal.project file specifies (which itself can of course use stackage or any other collection) including the solver/planner.

[ElvishJerricco]

No I don't think so. The point of using Nix with Haskell is to use Nix to define all your dependencies in a package set. Cabal should be doing no planning / solving, as that work is meant to be done by the nix expressions.

I would expect it to work exactly how it does currently with cabal build. Currently, if my nix-shell provides a GHC with all my Haskell dependencies, I can do cabal configure from within that shell to configure cabal to use the dependencies from that environment, and I can use cabal build to do a local build depending on that nix environment.

[dcoutts]

Ok, so that's a completely different requirement and it's incompatible with cabal new-build and project files. We cannot use new-build to get nix to install all the deps when nix uses its own choice of configuration of those packages because then there is no guarantee that the configuration matches up with what we need for this project.

The project file (+ solving & elaboration) determines the configuration of all the Haskell dependencies all the way down. It does not just pick versions but all details of the configuration (which it then hashes nix-style). We cannot arbitrarily substitute differently configured deps and expect it to work. The fact that the old build's nix integration code did this is no evidence that such a thing is possible. That approach was not deterministic and was easy to break for exactly these kinds of reasons.

[dcoutts]

There are three levels of integration I can imagine:

1. Just use nix to build non-Haskell deps, like ghc and C libs etc, just as we might try to do with RPM or DEB based systems. Ie just treat it like any ordinary dumb system package manager.

2. Generate nix expr(s) for the cabal project based on the exact config that the cabal new-build code would choose natively. This would work essentially by translating the new-build install plan into a nix expr. This would be a separate mode and would have no other CLI integration. It would be useful for doing CI for cabal projects via nix.

3. Do 2. above but also include CLI integration so that under the hood the new-* commands use nix to execute the translated plan, rather than executing the plan directly as they do now. This would also allow entering a nix shell with all deps in the right configuration, but it'd also allow just running cabal commands directly and it'd use nix to build non-local deps.

[ElvishJerricco]

To be totally clear, the point here isn't to use new-build for its Nix style build infrastructure. It's almost exclusively just needed as a means to build multi-package projects against a Nix environment, rather than any environment solved by Cabal. This is consistent with the workflow that is possible today with cabal build, which also uses Cabal exclusively for local incremental compilation and configuration, without any solving.

I also don't believe it should be incompatible. I can actually emulate the desired effect just by manually entering a nix-shell and running the Cabal commands in there myself. new-build is more than happy to use whatever GHC you throw at it, including the packages that come with it. And it will correctly fail when one of those packages is outside the constraints in the cabal files. The point of this issue is merely that --enable-nix should enter that Nix shell itself.

[ElvishJerricco]

We absolutely do not want Cabal dictating Nix expressions. This completely defeats the purpose of nixpkgs.

[dcoutts]

Ok, it's true that cabal can solve for "pinned" pre-installed packages such as the ones that are in the global package db. In principle one could try doing that with a very large pre-installed collection. It'll obviously completely change the solver results vs what you'd get normally, but if the solver finds a solution then it should indeed be ok.

So I take it back, it is plausible by effectively changing the environment in which we consider the project. Instead of being a GHC + core libs + hackage source packages environment, it'd be a GHC + nix hs libs (+ hackage?) environment. What cabal would need in this case is the information for all the pre-installed packages, but presumably before they are actually installed since we'd want cabal to tell nix to install them. Basically we'd need the nixpkgs collection to provide a database in the style of the ghc-pkg format. This would then be an input to the solver, much like the ghc-pkg global db is now.

So yes you can enter a suitable nix shell now and cabal will pick up those global packages, but if you want cabal to be able to pick/configure that shell then it needs the info up front about all those packages it can choose from. I don't know how the current cabal build code does that, I rather suspect that it does not.

[ElvishJerricco]

In this workflow, Cabal doesn't pick/configure that shell. You use Nix for this stuff to have that shell predefined via your Nix expressions. It's much like Stackage in that regard, in that your Nix shell pulls the packages you need from a predefined package set which is not solved by Cabal, but rather defined in a large Nix expression somewhere upstream (probably on GitHub in NixOS/nixpkgs). This should never ask Cabal for its build plan, and Cabal should never try to install anything. The Nix shell handles all of that.

[3noch]

To clarify, cabal2nix creates a nix expression that produces a real, legitimate ghc-pkg db. That db is then used as the "global" db from within the isolated nix-shell environment. cabal itself need not even be aware that any of this is happening.

[dcoutts]

Then I don't understand. If you're already in a nix shell pre-configured with the nixpkgs environment then everything already works, what is there to integrate?

[ElvishJerricco]

Not much =P In all the other commands, --enable-nix literally just does the normal thing, but inside the nix-shell. That's all this issue is about. cabal new-build --enable-Nix should do exactly what it does now, except inside the shell.

[dcoutts]

--enable-nix literally just does the normal thing, but inside the nix-shell. That's all this issue is about. cabal new-build --enable-Nix should do exactly what it does now, except inside the shell.

I'm sorry, I literally don't know what this means. What shell are you starting in? Is cabal starting a new shell instead of executing it's normal effect? Or is it executing its normal effect from within some new shell? How does it know what shell to start? Why do we need to be inside a new shell when you as a nix user can just enter the shell directly and run all cabal commands inside it?

For context: it's worth noting that I have no idea what cabal build --enable-nix does. I didn't implement it and I've never used it.

[ElvishJerricco]

Apologies for explaining this so poorly. I'll try to start from the beginning to provide maximum context for those of us who don't use Nix.

Nix is a programming language that you use to define build systems. Its main tenets are laziness and purity. Any given build (called a "derivation") is essentially two main components: the shell required to build it in, and the script to build it. This script is generally meant to be "pure," in that it is hopefully close enough to deterministic. Therefore, all the dependencies provided to a derivation's shell are also fairly deterministic.

It's usually considered best practice with Nix to avoid global installation of things, and instead to rely on building nix shells that provide you with all the tools you need to do your build. This way merely modifying the nix expressions you use does not require any lengthy installation process. The next time a tool or person enters the nix-shell, they will be given the correct, updated environment without even thinking about it.

With Haskell, you define a nix shell that depends on all the haskell dependencies you need. Then you enter the nix shell and use cabal from there. Now, there are some shorthands you can use to do this automatically: nix-shell . --run "cabal new-build". These work fine, and you can do all the same things with it. But on top of being tedious to always type, this is pretty bad for supplemental tools. Many tools have pretty good cabal support, but no nix support. If cabal would do the nix-shell part for you, any tool that runs cabal itself would work for free.

[dcoutts]

(BTW I'm quite familiar with the nix principles, I've read the papers and you may notice that cabal new-build is based on stealing ideas from nix: determinism, hashing to generate ids, nix-style store, multiple independent environments etc)

Ok, so one thing I don't get here is why don't you just enter the nix shell first and then run cabal or other tools that invoke cabal? Why would you want to start in some other shell and run nix-shell . --run "blah"?

So your "nix support" is effectively just a shell script that does nix-shell . --run $1?

[ElvishJerricco]

Unfortunately, that is not always up to me =P The most common example is editor tooling. Unless I want to always start my editor from inside a nix shell, and restart it after any change to any nix expression, most editor integrations will rely on calling cabal bare.

It is also tedious to have to keep track of whether I need to exit the nix shell before running my next cabal-related command.

[hvr]

I'm still trying to understand what we'd need to do beyond what can be accomplished by a shell-wrapper as has been pointed out that could be simply named cabal (while the real cabal-install is called cabal.real) that looks more less like

#!/bin/sh

exec nix-shell . --run "cabal.real $@"

[ElvishJerricco]

Scripts like that have all sorts of edge cases. For example, when that script is run when the working directory is not the project root directory (or worse, when the working directory isn't the project root and has a different shell.nix file).

[hvr]

Ok, let's assume (again, this is for me understanding the requirements) we already had cabal new-path --project-root with prints path of the project-root to stdout. Would that allow to capture the desired logic in an augmented shell-script (nix-shell $(cabal new-path --project-root) --run "cabal.real $@")?

[ElvishJerricco]

I'm still not convinced that no edge cases remain, though I admit I can't think of any. Regardless, as cabal-install evolves, it will surely grow new edge cases.

Plus, I don't really think this is sufficient. I was saving this for another issue, but I think you should be able to do some Nix configuration in the cabal.project file. Something like:

packages: a/
          b/
          c/

nix:
  enable: true
  options: --cores 1 --pure --arg myNixArg false

# shell.nix
{ myNixArg ? true }:

...

It seems useful to be able to give slightly different parameters for Cabal shells.

Also, I tend to think wrapper scripts are unfortunate hacks. Their prevalence in nixpkgs makes me queasy =P

[ElvishJerricco]

The other thing is consistency. If we're not going to support it in new-build, we ought to remove it from old-build. Judging by the reaction in the original PR, I'm thinking this would be a major letdown.

[hvr]

I see. But from what I gather, we'd need different modes of Nix support. The one you're suggesting, as well as the variant Duncan was originally talking about (which is the one that sounds intriguing to me personally - I'd like to see new-build be able to emit its install-plans as nix expressions). We should take that into account when designing the CLI & cabal.project syntax, so that those modes don't step on each others toes.

[ElvishJerricco]

I'm not sure anyone would really use that mode, considering a big reason to use Nixpkgs is to avoid solvers =P Anyway, I don't think that would affect the same things as the --enable-nix flag. That sounds like a different mode of the solver, not of new-build. Seems to me more like it ought to be a command that runs the solver and saves Nix expressions to files, which you could then import in your shell.nix and use with --enable-nix without changing the design of --enable-nix at all.

[3noch]

Having cabal generate nix derivations of any sort would effectively replace cabal2nix which is now a very deep part of the Haskell system on Nixpkgs. @ElvishJerricco is correct that the Nixpkgs ecosystem has zero interest in actually doing dependency resolution. Everything is static and completely deterministic. As soon as you enter non-deterministic land, Nixpkgs is the wrong tool for the job. In that way, having cabal generate nix derivations based on the solver would essentially be an impedance mismatch with Nixpkgs entirely.

At the same time, when a solver-based dependency resolver is what I want, I want to stop using Nixpkgs at that point and just use cabal as-is.

[3noch]

If it's any impetus to keep the --enable-nix functionality, this is a feature supported by stack so that tooling can remain simple and consistent. In friendly competition, it seems dropping this feature from cabal would only serve to push more people to stack and, well, we don't necessarily want that do we? ;)

[hvr]

I'm not sure anyone would really use that mode

At least I would! ;-)

Is there some in-depth description of Nixpkg I can read upon? I always assumed that Nix was powerful/flexible enough to naturally complement Cabal's nix-style builds in a seamless way; as opposed to traditional Linux distributions which force you to "flat" predetermined install-plans, and where cabal new-build effectively has to manage its own isolated environment, with the underlying distribution being agnostic to cabal (and the best we could do there, is make cabal somewhat agnostic of dpkg and rpm). Also, I don't understand what you mean by "completely deterministic", the solver is deterministic if you don't change its input; in fact a killer feature of cabal new-build is that it's more "deterministic" by being state-less regarding the package-db, i.e. we do away with the user-pkg-db, so there's one less (state-full) input.

Also, this isn't about dropping a feature (nor should this be about quickly adding features w/o proper design - merely because Stack has done so already - every time we do that, we risk adding technical debt and maintenance burden, and we're stuck with potentially inconvenient UI design choices), but about making sure this is done properly, so that the feature doesn't conflict with another feature I'd love to see in Cabal!

[dcoutts]

@ElvishJerricco the first use case I described is a real use case for people I work with, for their CI. They're using nix for reproducibility and integration with non-Haskell code not for the nixpkgs. They're using stack/cabal/hackage for solving the package sets. So as @hvr says, we should keep these different use cases clear so that we don't confuse things (e.g. with stuff in the project file).

[ttuegel]

What is the goal exactly for nix integration with new-build?

The only goal of the existing Nix integration is:

If we are doing a local build, and a Nix expression exists in the same directory as the .cabal file, then run cabal inside a nix-shell of that expression.

Discussion of anything else, in particular generating Nix expressions, is off-topic noise outside the scope of this issue.

What are the challenges involved in implementing this? Do you think it's a reasonable goal for someone like me who has never contributed to Cabal before?

Looking at the implementation for cabal repl, the integration requires wrapping entry points with the nixShell function. That aspect is relatively simple. I don't know how the new-build command works, but I seem to recall that it doesn't use the --builddir option. The temporary files we need for the nix-shell are stored relative to --builddir, so that would be a problem. Someone familiar with new-build should comment more.

[ip1981]

Nix integration is a joke, don't do it :) Stack failed with it. It is simply out of scope. After all, why not integrate with RPM or APT?

P. S. As well as with Docker.

[int-index]

Nix integration in Stack works well for me (on NixOS), I'd really like something similar in cabal.

[Gabriella439]

If people don't object I'll try to summarize the discussion so far as somebody who has used all three of cabal/stack/Nix extensively:

• Nix's Haskell support works similar to Stack, meaning:
  • Nixpkgs maintains a curated set of packages (analogous to a Stack resolver)
  • You can override a package within that package set to use a different version from Hackage (analogous to the extra-deps section of a stack.yaml file)
  • You can also add or override packages from GitHub or use a local source checkout of the package (similar to the packages section of a stack.yaml file)
  • Users typically prefer not to use Cabal's solver but can if they really want to (similar to Stack)
• Nix's Haskell support differs from Stack in the following way:
  • Stack forces cabal to use its pinned dependencies using a giant cabal.config file (like this one)
  • Nix forces cabal to use its pinned dependencies by generating a wrapped GHC whose global package database contains those dependencies
• Running all cabal-install commands inside of a nix-shell works with V1 commands and appears to work with V2 commands, too
  • I say "appears" because I haven't used V2 commands extensively in my own Nix+Haskell development, and I've only done some light testing
• The V1 commands support a nix: True ~/.cabal/config flag which allowed the same commands to work outside of a nix-shell
  • It sounds like a small thing, but it's extremely convenient for Nix users
  • It also reduces user error since Cabal's Nix integration will regenerate the nix-shell any time you run cabal configure (either explicitly or implicitly as part of another cabal command that detected a configuration change)
• The request here is to support the same nix: True configuration option for the V2 commands

Feel free to correct me if I got anything wrong

[gbaz]

One further issue is that cabal v2 commands will always fetch and install packages that aren't in the db into the store, so you can't ensure you're only using the "closed universe" of packages made available into the custom pkgdb that nix builds. There are workarounds that can/should be integrated into nix scripts. In particular, setting a custom cabal_config env variable (as per here: https://github.com/haskell/cabal/issues/5322) can be used to prevent cabal from installing additional packages from hackage.

[j6carey]

Following on the comments of @Gabriel439 and @gbaz :

When I enter a nix-shell and then use cabal new-* (v2 commands), build tools required by my .cabal file seem to be built by cabal even if they are available from the nix-shell environment. In particular, any patches applied by Nix when building those build tools have no effect. Using v1 commands does not seem to have the same problem.

[quasicomputational]

@gbaz, new in cabal-install 3.0, reject-unconstrained-dependencies allows you to restrict the solver to a closed set. It'd be a little bit cumbersome to repurpose for this, though - I guess you'd want to generate a cabal.project (or perhaps a freeze file?) with every package in the pkgdb that Nix creates.

[Ericson2314]

IMO https://github.com/input-output-hk/haskell.nix is the future, and a much better friend of Cabal and stack in particular, and any future work on nix-cabal integration should focus on that.

One goal in that regard is that Nix and Cabal should be able to hash files in the same way. I think the best thing to do is teach them both git tree hashes. Once we reach the point where hackage builds, patched hackage builds, and local builds are all cached with comparable keys, things will start to move smoothly.

[neongreen]

The only goal of the existing Nix integration is:

If we are doing a local build, and a Nix expression exists in the same directory as the .cabal file, then run cabal inside a nix-shell of that expression.

Until this is solved in Cabal proper, you can use a workaround: nix-cabal.

Installation:

curl https://raw.githubusercontent.com/monadfix/nix-cabal/master/nix-cabal -o $HOME/.local/bin/nix-cabal
chmod u+x $HOME/.local/bin/nix-cabal

Then whenever I want a project to use nix-cabal, I create a .dir-locals.el file in the project root and Emacs picks nix-cabal up:

;; .dir-locals.el

((haskell-mode
  . ((haskell-process-type . cabal-new-repl)
     (haskell-process-path-cabal . "nix-cabal"))))

In the shell, you just use nix-cabal foo instead of cabal foo, although you'll suffer the overhead of entering the nix-shell every time.

---

Another solution is lorri. It runs a background daemon that keeps nix-shell warm, and uses direnv to automatically enter nix-shell whenever you are in the project directory. It is somewhat finicky, though.

[jneira]

I think facts has demonstrated nix integration in v2 build is at best a very low priority enhancement. This issue has a very interesting discussion on the topic but i would close it if you all dont mind. If anyone still thinks the nix integration should be done please raise a hand.

[Ericson2314]

Last I checked (years ago) Cabal's (and Stack's) Nix integration wasn't really what I wanted as a Nix user anyways, so I don't mind if they whither away and are replaced with something more proper later.

[rnhmjoj]

Personally I've defined cabal = nix-shell --run "cabal $@" and this seems to be doing what I need, mostly. I still think it would be nicer to have this, so one can simply add nix: True to cabal/config.

[Ericson2314]

Mmm while that is better than nothing, it is so far from adequate Nix support it feels like false advertising to call that nix: True to me.