haskell: All boot libraries get added to pkgdb

[kirelagin]

I believe, this issue was introduced in #40996.

The essence of the issue is that when setupHaskellDepends is non-empty this code kicks in and, unfortunately, ghc is sitting in pkgsBuildBuild. Its package.conf.d directory has configs for all boot libraries and they all get copied, resulting essentially in all those boot libraries being added to the pkgdb when this was not intended.

One way to get around this would be to fix #39646 (sorry that I bring this up! 🤷‍♂️). Another would be to be more careful when copying things from pkgsBuildBuild and make sure to iterate only over Haskell dependencies (well, since ghc is a Haskell package as well, it is not that easy to tell the difference).

@angerman @Ericson2314 @peti (most likely this affects stackage2nix as well, so @4e6 FYI)

---

Now some motivation. I discovered this while debugging strange version conflicts in a package built with nixage (product placement!). In my specific case the problem was the following:

1. One of the boot packages is overridden with a newer version.
2. When you do this, you have to override all boot packages depending on it so that they see the new version. I work this around by overriding them with the same versions that are shipped with ghc, but take them from Hackage.
3. Those boot packages from point 2 end up being added to the pkgdb twice: first from ghc, second from my override.
4. Now there are two instances of the same package in pkgdb with equal version. I don’t know how Cabal chooses which one to use, but in my case it is choosing the one that comes with ghc, meaning that it will depend on the boot version of the package from point 1, which eventually leads to an error, because everyone else uses the newer version.

[peti]

@Ericson2314, do you have any opinion on this issue?

[kirelagin]

I think there are two problems currently that make things difficult:

1. Boot libraries are not first-class in nixpkgs, that is, it is impossible to see them in function arguments and handle separately.
2. GHC, even when it is meant as a build-tool (i.e. compiler), is treated by the pkgdb population logic as a Haskell library.

So, here are my ideas:

1. In order to fix point 1 what we could do is, instead of null-ing out core packages, replace them with fake ones, that will contain just a single package config file. This way boot packages will be actual derivations, not nulls, and it will be possible to add each of them separately to pkgdb. (Currently it is only possible to add all of them at once, because they all are in ghc.)
2. Separate GHC the compiler and GHC the library. Well, this will actually be handled automatically if we replace haskellPackages.ghc with a fake library, created as described earlier, which will contain only one package config file. The question is what will break.

3. (a) Then we can integrate some weird hack into the bash part of the generic builder which, when iterating over $pkgsBuild*, will be able to detect ghc the compiler and not copy package configs from it.

   (b) Or we can simply remove package config files from lib/ when building ghc the compiler.

   (c) Or we can rewrite the generic builder in such a way that it does not rely on pkg* variables being set by someone else, but instead directly work with haskellBuildInputs. I am not sure, but I think the reason pkgs* variables from setup.sh are being used is that they handle propagation. I bet something can be done with it. E.g. one option would be to store the pkgdb each package was built with in nix-support and easily propagate via it.

What do you think?

[Ericson2314]

I think boot libraries are the root of almost all build systems evil. I'm very eager to get rid of them.

[angerman]

Just a quick note: we currently can not replace all libraries that GHC ships with. Some are non-reinstallable. Ideally GHC would just be a loose set of dependencies where each one can be swapped out. This sadly won't work completely and there are some package which simply must come from the set of shipped packages. Cabal maintains a list here:

https://github.com/haskell/cabal/blob/a655942bdbfb4e5e0f4eba1bf23d400f5e2a617a/cabal-install/Distribution/Solver/Modular/Solver.hs#L136-L147

There are some nasty issues with reinstalling dependencies of those, as you end up with duplicate packages in your dependencies by then. So what you really want is to blacklist the packages mentioned above together with their transitive closure.

I use some similar, but rather crude hack in my stackage package set generator:

https://github.com/angerman/stackage.nix/blob/bfe36ce019846af7899c05e350bd8f92ee2d9136/package-set.nix#L10-L12

[Ericson2314]

@angerman it's my understanding that the only impedement to building those separately are see cyclic dependencies (the wired-in stuff, etc). Even if there's exactly one version * configuration that works with GHC, it would be nice to build it separately. Am I missing something?

[angerman]

@Ericson2314 probably not all of them, but there are some codegen api/abi dependencies as well. So while I believe reinstalling the identical version might work, upgrading them in any form would be rather destructive.

Also not all of them can actually be built without issues outside of the ghc tree. I've recently tried to build integer-gmp outside of ghc, and failed pretty badly. That however is nothing that couldn't be fixed I believe.

[kirelagin]

Well, I’m not suggesting to build them separately (at least not at the moment), I am suggesting to cut the ghc output into individually-installable “fake” libraries, which will consist merely of a config file. Won’t this work?

[angerman]

@kirelagin that will most certainly work. You could even take the archives/shared libs/.hi files out of ghc, and place them into separate packages (multiple-outputs?). As long as you retain the path's in the config files, you can install those in any package-db you like.

[kirelagin]

@angerman

multiple-outputs

Yes, that is what I was thinking.

You could even take the archives/shared libs/.hi files out of ghc

I’m not sure why I would want to do this, since the paths in the config files will be pointing into the ghc output. If I take the libs and stuff into separate outputs, I’ll have to patch the config files, and I don’t see any benefits in doing so.

[kirelagin]

I just need to somehow make sure that there is a runtime dependency between each base-library output and the main ghc output 🤔.

UPD: apparently, just having a path in a text file is enough for nix to notice the dependency 👍.

[peti]

The problem is that https://github.com/NixOS/nixpkgs/blob/95a8cb3ade1ad0e2649f0f3128e90c53964af5e1/pkgs/development/haskell-modules/generic-builder.nix#L277-L286 takes the dependency information from the wrong place. The list of setup dependencies is readily available from setupHaskellDepends, but instead of using that accurate information, the loop instead creates an environment using information from Nix's generic builder which, unfortunately, throws ghc into that mix. That code in the generic builder ought to be fixed.

[kirelagin]

@peti Right, I think I asked about this in one of the other haskell builder related PRs but didn’t get a definite answer. My current theory is that it is done this way because setup.sh handles propagation.

[peti]

Yes, I was lazy when I wrote that code originally in the generic builder, and now that choice has come back to bite us. :-( The proper solution would have been to compute all those paths statically at evaluation time and to use lib.closePropagation directly, like the ghc wrapper does.

[Ericson2314]

closePropagation needs an overhaul to match the findInputs changes I made for cross.

Without or with those changes, we can solve this problem by making generic-builder.nix only pick up base libraries from @kirelagin's base library outputs I'd think.

[nh2]

Probably tangentially related, mentioning for completeness: 387c513d124694e44057446006b524e4ed367f35.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.