input-output-hk / haskell.nix

Alternative Haskell Infrastructure for Nixpkgs
https://input-output-hk.github.io/haskell.nix
Apache License 2.0
555 stars 241 forks source link

build-tool-depends executables don't appear in shellFor #839

Open ocharles opened 4 years ago

ocharles commented 4 years ago

I want to use tasty-discover, which I have added to my .cabal file as:

  build-tool-depends: tasty-discover:tasty-discover

As described in https://github.com/input-output-hk/nix-tools/issues/95, this adds a build-tools entry in the cabal-to-nix expression, which is used in setup-builder.nix to form executableToolDepends. However, it doesn't propagate correctly into the shell produced with shellFor. For build-tool-depends to work, we need a tasty-discover entry in the ghc-pkg database that also has the tasty-discover executable. As things currently stand, there is no entry at all. We can use additional = pkgs: [ pkgs.tasty-discover ], but that will only add a tasty-discover to the package database as a library - no executables will get included.

michaelpj commented 4 years ago

Related/dupe: https://github.com/input-output-hk/haskell.nix/issues/760 https://github.com/input-output-hk/haskell.nix/issues/231

ocharles commented 4 years ago

Ah, thanks!

rvl commented 4 years ago

I think I know how to fix this now, thanks @ocharles.

ocharles commented 4 years ago

@rvl Oh, fantastic! Do you need any help with anything? Happy to test anything, if that helps!

JBetz commented 4 years ago

A similar issue seems to happen if your shell dependencies have build tool dependencies.

E.g., cardano-transactions has a test suite with a build tool dependency on cardano-tx, and when I try to create a shell on a cabal project that uses it, it errors with:

error: Neither the Haskell package set or the Nixpkgs package set contain the package: cardano-tx (build tool dependency).
If this is a system dependency:
You may need to augment the system package mapping in haskell.nix so that it can be found.
If this is a Haskell dependency:
If you are using Stackage, make sure that you are using a snapshot that contains the package. Otherwise you may need to update the Hackage snapshot you are using, usually by updating haskell.nix.

Not sure if the same thing happens on stack projects, but I would presume not considering that cardano-transactions uses stack.

michaelpj commented 3 years ago

I spent a little bit of time looking at this.

Firstly, I noticed that we don't pass build-tools into the fake component that we use to create the wrapped GHC for shellFor. Easy enough to fix.

But then it still doesn't appear in the package db. It turns out that makeConfigFiles doesn't use build-tools at all. So... how does anything work? My guess is that the Setup.hs interface to cabal doesn't insist on build-tool-depends being in the package db, and so we get away with just providing the executable. But probably we should register them, I'm just not sure how right now.

Ericson2314 commented 3 years ago

Yeah I don't think they should be registered in the same package db, especially if package DBs are to be per machine for cross. When you need a library, you usually need lots of information beyond the machine code (unfoldings for inlining, etc.) but for an executable dependency, one just needs to be able to run the thing, and being agnostic to how the sausage is made is probably better for sake of determinism / caching.

michaelpj commented 3 years ago

Yeah I don't think they should be registered in the same package db, especially if package DBs are to be per machine for cross.

Well, not having it in the package db confuses cabal, so I don't see how we can avoid it (if we want this usecase to work!).

As far as cross goes, obviously the build-tools should be in the native package db in the glorious future once we've got a separate native package db. Then I think there's no real problem with them being there.

matthewbauer commented 3 years ago

Note that this really isn't specific shellFor, but specific to v2 cabal. You can always use v1-* cabal commands to get the old behavior (see SolveExecutables in cabal code). I think a reason this is coming up recently is that cabal 3.4.0.0 made v2 the default.

The reason this is really difficult is that both Nixpkgs & haskell.nix use the Setup.hs method of configure / install, which doesn't install a .conf file for executable components. I'm not sure if it makes more sense to patch Cabal so Setup.hs method installs executables correctly, or try to switch to the new install method.

See Cabal comment for reference:

build-tools dependencies NB: Only include these dependencies if SolveExecutables is True. It might be false in the legacy solver codepath, in which case there won't be any record of an executable we need.

https://github.com/haskell/cabal/blob/master/cabal-install-solver/src/Distribution/Solver/Modular/IndexConversion.hs#L348-L357

Ericson2314 commented 3 years ago

Thanks for digging into that!

fgaz commented 2 years ago

As far as cross goes, obviously the build-tools should be in the native package db in the glorious future once we've got a separate native package db.

v2 cabal-install recently got a --package-db flag that an be used to specify additional package dbs, though I'm not sure if it can actually help with this issue. Here's the pr for refernece: https://github.com/haskell/cabal/pull/7676

tscholak commented 2 years ago

For the record, this is also an issue for llvm-hs, which needs the hsc2hs executable: https://github.com/llvm-hs/llvm-hs/blob/eda85a2bbe362a0b89df5adce0cb65e4e755eac5/llvm-hs/llvm-hs.cabal#L74.

codygman commented 2 years ago

The latest state of this issue seems to be: https://github.com/input-output-hk/haskell.nix/issues/1367#issuecomment-1207454622

michaelpj commented 2 years ago

No, I think that's unrelated. This isn't anything to do with source-repository-packages. This is specifically about how build-tool-depends works.

L-as commented 2 years ago

It seems like this is a duplicate of #231?

michaelpj commented 2 years ago

This one is more general, yes the other is a dupe.

parsonsmatt commented 2 years ago

Is there any work that can be done to help solve this problem? We're trying to migrate to haskell.nix but this is a problem for us. We'd be happy to contribute fixes or consulting time to the issue.

ParetoOptimalDev commented 2 years ago

Is there any work that can be done to help solve this problem? We're trying to migrate to haskell.nix but this is a problem for us. We'd be happy to contribute fixes or consulting time to the issue.

I haven't been able to look into this yet, but I really hope you are able to push this front forward.

My primary selling point for haskell.nix has been "You can avoid knowing nix and just update your cabal.project/package.cabal file but get cross compilation, reproducibility, docker containers, etc". @parsonsmatt I see in your linked issue your team is wanting the same: Use cabal.project as a source of truth.

Partial cabal.project support breaks this sellling point. I hope to deep dive into this issue and figure out haskell.nix well enough to solve some of these problems myself in the future, but I have other high priority things blocking it sadly :cry:

andreabedini commented 1 year ago

After commenting on the linked cabal issue, I came here to say this is a short-coming of shellFor and should be fixed in haskell.nix, but then I noticed I do get tasty-discover in my devshell!

❯ nix develop --command which tasty-discover
/nix/store/6wmdh78kw1qzb4kgl6nchpsyhlxsds25-tasty-discover-exe-tasty-discover-5.0.0/bin/tasty-discover

Did the issue solve itself? Or maybe the issue is that, inside the shell, cabal will recompile tasty-discover anyway? (which is related to https://github.com/input-output-hk/haskell.nix/issues/1662).

JackKelly-Bellroy commented 1 year ago

I don't think we're all the way there. I too am seeing tasty-discover show up in my shell now, but I tried deleting my ~/.cabal/store and rebuilt a tasty-discover-using project, and cabal still wanted to build tasty-discover.

andreabedini commented 1 year ago

@JackKelly-Bellroy thanks for double checking! Yes, unfortunately I would expect cabal to rebuild it (issue I have been discussing in the cabal repo). At least they should be exactly the same version; if not, it's a bug in haskell.nix we need to fix.

ScottFreeCode commented 1 year ago

FWIW this is an underlying nix-shell + cabal v2 problem and not just a haskell.nix thing – it's also possible in general to use cabal as the source of truth with nix's callCabal2nix and save haskell.nix for when you truly need its more complex features.

I have a straightforward reproduction without iohk's/iog's haskell.nix, here: https://gist.github.com/ScottFreeCode/c1adb5681c5b61c61373615f63a4a7d2

I've collected GitHub issues relating to this problem here: https://gist.github.com/ScottFreeCode/ef9f254e2dd91544bba4a068852fc81f

@andreabedini Here's another one you checked in on recently that appears to be this same issue.

toonn commented 3 months ago

I'm observing different behavior from that described here. tasty-discover does show up on my PATH and cabal-install doesn't seem to rebuild it but instead it rejects the installed tasty-discover.

> type -p tasty-discover
/nix/store/z4bhg9m76k78gkln10smq7bv9bph0ym7-tasty-discover-exe-tasty-discover-5.0.0/bin/tasty-discover
> cabal test --enable-tests
Warning: No remote package servers have been specified. Usually you would have
one specified in the config file.
Resolving dependencies...
Error: [Cabal-7107]
Could not resolve dependencies:
[__0] trying: taskell-1.11.4 (user goal)
[__1] trying: taskell:*test
[__2] next goal: taskell:tasty-discover:exe.tasty-discover (dependency of taskell *test)
[__2] rejecting: taskell:tasty-discover:exe.tasty-discover-5.0.0/installed-68WXL8qsKSKFFqrGegwKHL (does not contain executable 'tasty-discover', which is required by taskell *test)
[__2] fail (backjumping, conflict set: taskell, taskell:tasty-discover:exe.tasty-discover, taskell:test)
After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: taskell, taskell:test, taskell:tasty-discover:exe.tasty-discover, tasty-discover
Try running with --minimize-conflict-set to improve the error message.

Haven't been able to come up with a workaround so far.

EDIT: Workaround proferred by sclv is to provide tasty-discover on the PATH in another way and remove the build-tool-depends from the cabal file. This does break the single-source-of-truth that we're all after though.