Open michaelpj opened 1 year ago
@andreabedini are these included in the plan.json?
(Ouch, I had missed this ticket 🫣)
Yes, extra-packages should be in plan.json as they are part of the plan like a local package. Not entirely sure how to identify them as "extra" but maybe we should not have that problem.
In #1861 I end up saying
shellFor and cabal repl need to agree re what packages to have in scope.
It turns out they do agree, extra-packages
are not in scope in cabal repl
.
❯ cat cabal.project
-- hello depends on aeson
packages: ./hello.cabal
extra-packages: lens
❯ cabal exec -- sh -c 'cat $GHC_ENVIRONMENT | grep "\(aeson\|lens\)"'
package-id aeson-2.2.0.0-dcd0d2b64774f31f4ee1a9c03fbf6eafaf340534276728e158adeceb8fad54f0
aeson is in scope as a dependency of hello but lens is not.
Therefore, if the mantra is to do what cabal does, haskell.nix does the right thing here.
@andreabedini I'm not sure checking the environment cabal provides is an accurate test. cabal exec --help
says:
Compiler tools will be configured to see the same subset of the store that builds would see
And I assume that subset is what $GHC_ENVIRONMENT
is showing.
The extra-packages
aren't dependencies of any build target (and that's the whole point), so they're not going to be in "the same subset of the store that builds would see". They're just supposed to be installed, which makes them accessible to bare ghci
(and usable in cabal repl
by using :set package
commands).
@cumber You are right. My view of what shellFor should do is a little superficial.
I had missed some facts about how cabal exec
works (see below) and forgotten that cabal exec
also adds any executable in the plan to $PATH (check by adding e.g. cabal-plan
as an extra-package).
I am happy to take back my statement that "shellFor and cabal repl need to agree re what packages to have in scope" but then my question becomes, if it's not cabal exec
or cabal repl
, what should use to model shellFor
behaviour?
Should we just say that shellFor provides a environment with all project packages and executables in scope? As shown below cabal exec
kinda tries to do this but not very reliably.
cabal exec
❯ cabal exec -- sh -c 'cat $GHC_ENVIRONMENT' | grep package-db
clear-package-db
global-package-db
package-db /home/andrea/.local/state/cabal/store/ghc-9.4.6/package.db
package-db /home/andrea/Scratchpad/haskell-nix-issue-1861/dist-newstyle/packagedb/ghc-9.4.6
notice the entire cabal store is passed in the environment.
Therefore, the following works
❯ cabal exec -- ghci
Loaded package environment from /home/andrea/Scratchpad/haskell-nix-issue-1861/dist-newstyle/tmp/environment.-7203/.ghc.environment.x86_64-linux-9.4.6
GHCi, version 9.4.6: https://www.haskell.org/ghc/ :? for help
Loaded GHCi configuration from /home/andrea/.ghci
λ> :set -package lens
package flags have changed, resetting and loading new packages...
picking up the lens that I have in the cabal store
❯ ghc-pkg --package-db=/home/andrea/.local/state/cabal/store/ghc-9.4.6/package.db list lens
/home/andrea/.local/state/cabal/store/ghc-9.4.6/package.db
lens-5.2.2
This seems a bit incidental and fragile to me because nothing is guaranteeing that there is at least one lens in the cabal store is not guarateed (and who knows what happens if there are more than one! edit: using -package
will let ghc guess while using -pacakge-id
you can specify which one you want; tl;dr: -package lens
does not guaratee you get the one in the plan).
Starting for a clean slate (no store and no dist-newstyle), cabal exec
does not trigger any building and just fails to find the store packagedb.
❯ cabal exec -- ghci -package lens
Resolving dependencies...
Loaded package environment from /home/andrea/Scratchpad/haskell-nix-issue-1861/dist-newstyle/tmp/environment.-11124/.ghc.environment.x86_64-linux-9.4.6
GHCi, version 9.4.6: https://www.haskell.org/ghc/ :? for help
ghc-9.4.6: can't find a package database at /home/andrea/.local/state/cabal/store/ghc-9.4.6/package.db
Doing cabal build
alone won't build the extra-packages
, you need to build them separately.
❯ cabal build
Build profile: -w ghc-9.4.6 -O1
In order, the following will be built (use -v for more details):
...
❯ cabal exec -- ghci -package lens
Loaded package environment from /home/andrea/Scratchpad/haskell-nix-issue-1861/dist-newstyle/tmp/environment.-19906/.ghc.environment.x86_64-linux-9.4.6
GHCi, version 9.4.6: https://www.haskell.org/ghc/ :? for help
<command line>: cannot satisfy -package lens
(use -v for more information)
❯ cabal build lens
Build profile: -w ghc-9.4.6 -O1
In order, the following will be built (use -v for more details):
...
Completed lens-5.2.2 (lib)
❯ cabal exec -- ghci -package lens
Loaded package environment from /home/andrea/Scratchpad/haskell-nix-issue-1861/dist-newstyle/tmp/environment.-22911/.ghc.environment.x86_64-linux-9.4.6
GHCi, version 9.4.6: https://www.haskell.org/ghc/ :? for help
Loaded GHCi configuration from /home/andrea/.ghci
λ>
@andreabedini Yeah, you seem to be right. Cabal doesn't seem to do anything to make extra-packages
installed, it just picks them up if they are in the store. I suppose that's deliberate, because the whole point is that building the project shouldn't depend on the extra-packages
(otherwise they'd just be dependencies). Arguably cabal repl
should? I might file an issue with them and see what they say.
It does actually do solving with them though, they're not simply ignored. If you are in a folder with a cabal.project
containing extra-packages: raw-strings-qq
and then you manually cabal build raw-strings-qq
, it installs version 1.1
. Whereas if your cabal.project
lists extra-packages: raw-strings-qq ^>= 1.0
, then it installs version 1.0.2
.
Maybe this is a bump that can't fully be reconciled.
My mental model of how Haskell.Nix is supposed to work is that it sets up a "pre-installed" cabal store in the nix store so that cabal would never need to build/install any other packages. So I had expected that extra-packages
would simply be included in that set.
But of course, if I were releasing a finished product build based on Haskell.Nix, I then wouldn't want those extra-packages
to be dependencies of the final builds. Only in the dev shell. But if the dev shell and the final builds are produced using the same nix derivation for the cabal store, that can't work. In that case using the nix side additional
to control whether they're actually installed or not seems reasonable.
(Actually my initial instinct was to try to add in extra packages to the build plan from the nix side, and was told "As ever, the Haskell.Nix Way is to configure what you want with cabal")
It is unpleasant that using extra-packages
alone completely breaks the REPL with little explanation, which wouldn't happen when working with cabal directly. (It does happen when you have a package-db but don't have an index from hackage at all, but that's an extremely unusual state when working with cabal directly). I don't know if anything can be done about that; it seems that Haskell.Nix would either need to filter the extra-packages out of the build plan entirely, or include them in the built store, or have mechanisms for explicitly deciding which of those two you want (additional
provides the explicit include; is there anything for the other option?). Or else just say that cabal projects using extra-packages
aren't fully supported.
Ideally I would want extra-packages
to just magically be available in shellFor
environment, but not when the package is finally built (e.g. building an executable output with nix, or including the library as a dependency of some other Haskell project). I don't know whether that's possible though.
Either way, thanks for your time looking into it!
@cumber Let me be clear: I needed to understand your expectations and cabal's behaviour but I didn't mean to argue that your expectations were wrong :)
It does actually do solving with them though, they're not simply ignored. If you are in a folder with a
cabal.project
containingextra-packages: raw-strings-qq
and then you manuallycabal build raw-strings-qq
, it installs version1.1
. Whereas if yourcabal.project
listsextra-packages: raw-strings-qq ^>= 1.0
, then it installs version1.0.2
.
Yes, extra-packages
are part of the plan (with their own constraints). They don't seem to be included in the default build targets though (even cabal build all
does not build them).
Ideally I would want
extra-packages
to just magically be available inshellFor
environment, but not when the package is finally built (e.g. building an executable output with nix, or including the library as a dependency of some other Haskell project). I don't know whether that's possible though.
TBH I think this is fair expectation. Once we understand that we are not modelling shellFor
after either cabal exec
or cabal repl
we are free to decide what we want shellFor to be :-).
I believe it should not be too hard to have extra-packages pre-built in the devshell (and only in the devshell). Note that their presence in cabal.project will influence the plan though, as it does with cabal.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Otherwise you get a shell where cabal is unhappy: https://github.com/input-output-hk/haskell.nix/issues/1861#issuecomment-1454329601