No clean way of adding non-Hackage and non-Stackage Haskell dependencies

[L-as]

See #1339.

That PR is an attempt to enable this functionality, however, it's a hack, just like the package: ${pkg} approach @angerman proposes. Like @michaelpj wrote, we want a clean way of being able to include external dependencies, perhaps something like the following:

{
  pkg-def-extras = hackage: {
     myPackage = callCabalToNix ./myPackage.cabal;
  };
}

This is also relevant for #1262.

[michaelpj]

Can you explain why source-repository-package doesn't do what you want?

[L-as]

source-repository-package doesn't allow anything not fetchable by Cabal, i.e.

• Non-git
• Private repository
• Source generated by Nix

In addition, it's a pain, and not easy to manage when you essentially need to have your own package index. E.g. in the Plutus ecosystem we do the whole copying cabal.project thing because we need to make sure we use the same version of e.g. cardano-prelude as upstream. This is all a huge PITA.

[michaelpj]

These are all problems with Cabal, rather than haskell.nix IMO.

E.g. in the Plutus ecosystem we do the whole copying cabal.project thing because we need to make sure we use the same version of e.g. cardano-prelude as upstream. This is all a huge PITA.

The correct solution is for us to stop using a massive web of source-repository-packages and instead use a proper package repository, either Hackage or an internal one. Again, not a haskell.nix problem IMO.

[angerman]

an internal one

😬 This would become one... but we should know in a few days, thanks to https://github.com/input-output-hk/hackage-overlay-ghcjs

[L-as]

I don't see how such a package repository would work without something like hackage.nix specialised to it.

[angerman]

@L-as i share similar concerns. The storage backing is different though. I'll see over the next few days.

[L-as]

I am a Nix zealot, so I am quite biased, but I imagine you could define the repository in pure Nix, such that it works (only) with exactDeps = true;.

[L-as]

@michaelpj This is another issue with source-repository-package: exactDeps doesn't work with it unless you don't have it in the shell and only while building the shell.

[michaelpj]

I am a Nix zealot, so I am quite biased, but I imagine you could define the repository in pure Nix

IMO haskell.nix is a project for Nix-ifying cabal projects, not a fancy Nix-based Haskell builder that goes beyond cabal.

[L-as]

I don't think that's just your opinion, that's the state of haskell.nix. I do concede that what I ultimately want is not haskell.nix, but something much more lightweight that rather than emulating what Cabal does and make it reproducible, presents a different Nix-native workflow, like I told you in London, however, I still believe that exposing the ability to use a custom Hackage essentially would not be very hard, and now that I think about it, it's probably the approach I should be going for.

[angerman]

Allowing custom hackage repos is certainly something I wish we already had. It's also the way cabal handles this.

Why do we focus so much on cabal? Because that's our ground truth. We don't (and can't: Windows, BSD, ...) force everyone to use nix, this the repos should work fine without.

But we do want nix in CI and for cross compilation, and ...

So if it can be expressed in cabal, Haskell.nix should just take it.

It's the same reason why cabal.project.local is my preferred interface when possible. It's the same strategy the cabal uses. And it ensures we get the same behavior.

One thing where Haskell.nix deviates from cabal is that we don't support multiple plans well. Hence the nonReinstallabaldPkgs madness :-(

[michaelpj]

Custom hackages already work, you just have to make them yourself: https://github.com/input-output-hk/haskell.nix/pull/535

[angerman]

Ohh I completely forgot about that PR! 🤦‍♂️

[brainrake]

Here is a prototype for using extra-hackages and extra-hackage-tarballs to add dependencies by path. https://github.com/mlabs-haskell/haskell-nix-extra-hackage/tree/0a1061b6a9da11da045c543f13ce32af9897ac74 Possible additions to haskell.nix worth considering:

• automatically add extra-hackages if extra-hackage-tarballs is provided but extra-hackages is not
• new extraHackagePackages argument to *project that does it all

[L-as]

@angerman @michaelpj Forgot to note this, but do note that pkg-def-extras can not be used for this, as it doesn't effect plan generation, hence the IFD will fail and complain that the packages are unknown, and no plan will be generated.

[L-as]

We had some issues with the above too, due to implicit ordering of hackages based on their names, which lead to Hackage taking precedence over the custom Hackage, meaning the compact-map IOG package couldn't be used, until we switched the names internally.

[michaelpj]

We have a working prototype of using a custom Hackage inside IOG. It's just a matter of getting buy-in and doing it now.

[L-as]

@michaelpj That would really be great. Do you plan to do something like the hackage.nix repo or something else?

[L-as]

Also, I think we can remove pkg-def-extras, since it has no use AFAICT?

[michaelpj]

Do you plan to do something like the hackage.nix repo or something else?

Better than that: the cabal repo itself is in a git repository, so you just pull it in as a flake input and do hackage-to-nix to it on the fly, so it's even simpler.

[ilyakooo0]

For the sake of completeness: I solved a similar problem here: https://github.com/ilyakooo0/haskell-nix-extra-hackage

[brainrake]

version with separate hackage per package, to reduce rebuilds https://github.com/mlabs-haskell/haskell-nix-extra-hackage/tree/separate-hackages