search

haskell / cabal

Official upstream development repository for Cabal and cabal-install
https://haskell.org/cabal
Other
1.62k stars 691 forks source link

Teach `cabal.project` about "dev dependencies" #6952

Closed michaelpj closed 2 years ago

michaelpj commented 4 years ago

Many build tools have the concept of a "dev dependency". These are dependencies which are managed by the build tool, but which are not used by any of the actual build products. Instead, they are there to provide tools that developers of the project will use.

Why have such things managed by the build tool? Can't users just install them separately? There are a few advantages, such as convenience and consistency between developers, but I think there is one that is very important in our case: the build tool may have information that makes it much easier for it to get the "right" tool. For example, many development tools themselves expect to interact with the language toolchain and build tool. It can therefore be important to pick a tool version that's compatible with the toolchain that you're using.

This is becoming very pertinent in the Haskell tooling ecosystem as more things depend on the ghc library. A tool which depends on ghc, like haskell-language-server, must be built with a version of ghc that exactly matches the compiler the user is using for a particular project (which may indeed differ between projects!). As it stands, this creates a lot of work for users and developers. Users must carefully manage their tool installations to make sure they have precisely the right version of the tool installed for the project in question.

But we already have a tool that has all the information about what version of ghc you are using and how that affects what versions of haskell-language-server you can use: namely cabal!

So much for the motivation, what am I concretely suggesting?

I don't think the dev-tools build should include constraints from the components in the project - the implicit constraints on ghc, Cabal etc. should be enough.

Partial duplicate of https://github.com/haskell/cabal/issues/5588, but I'm making quite a different case, which I think is much more compelling than just convenience.

Many of the other build tools which have "dev dependencies" also use them for things that have better solutions in cabal, e.g. for build tools, test dependencies etc. I think there's still a useful role for dev dependencies that are really for developers only.

A short list of things I expect people might want to put into dev-tools:

One could even imagine people who build their Haskell with alternative systems like shake might use a cabal.project to get the shake binaries in the first place.

michaelpj commented 4 years ago

I realised it's worth saying something about why consistency matters.

Some people like to format their codebase with stylish-haskell or remove all lints with hlint. Some people like to enforce that these conditions are met. But "is this codebase formatted correctly" also depends on the version of stylish-haskell you use. So your entire team needs to have the right version installed.

Just being able to write dev-tools: stylish-haskell:exe=2.0.0 or whatever would solve this quite effectively.

michaelpj commented 4 years ago

To be clear what I meant about the command, I imagine something like cabal dev-install or cabal install --dev would do it, but I don't really know enough about how cabal install works to comment sensibly at the moment.

phadej commented 4 years ago

I think this can be implemented as out-of-cabal-install prototype first to figure out the details.

phadej commented 4 years ago

In general I guess this is fine, yet why don't you use Nix? what if there is some dev-dependency which is not on Hackage, or not written in Haskell at all?

For stan you need same GHC, yet for HLint you don't want to recompile it with different GHCs (you really don't, it takes so long, that you'll lose interest in the project before it completes). There are details to be figured out.

michaelpj commented 4 years ago

I do use Nix! However:

For sure, you're still on the hook for non-Haskell dev dependencies, which is why I think this isn't a compelling feature if we're just talking about convenience of installation. But for me the ghc dependency issue is really quite annoying, and I think could be solved uniquely well by cabal.

I wouldn't mind compiling multiple versions of hlint for different projects. It's slow and annoying, sure, but it's much less annoying than having the wrong hlint. But yes, hlint is a weak case compared to the tools with ghc dependencies.

OOI, how would you go about solving this out-of-cabal-install? Don't you need access to the solver etc.?

michaelpj commented 4 years ago

I think haskell-language-server is a nice one to consider, because:

phadej commented 4 years ago

I will believe when I see a PoC. And as I see it, it can be done as an external cabal-install-dev-tools executable.

phadej commented 4 years ago

Side note: I really hope that GitHub discussions will soon be available for everyone. This is not an issue, this is an open-ended discussion.

This is wrong place for this kind of discussions. But we don't have better one.

michaelpj commented 4 years ago

Happy to close and take it elsewhere, but I do think it's useful to have something for people to find if they have a similar idea and I don't know where they would look except here.

phadej commented 4 years ago

See https://github.com/haskell/cabal/issues/6445#issuecomment-654944587

From what I see, the decision is made, so I'm saying this with the hope to find a better way to stay on top of these discussions and contribute in a timely manner when and where they happen.

alexbiehl commented 4 years ago

FWIW we have extra-packages in the cabal.project file. In principle it'd allow you to do cabal run stylish-haskell if you had extra-packages: stylish-haskell. Unfortunately it needs a bit of tuning in the target selector to fully work.

alexbiehl commented 4 years ago

I traced the target selector code and found that if you use fully qualified syntax like 

$ cabal run :pkg:stylish-haskell:exe:stylish-haskell

It happily starts building stylish-haskell but fails with 

cabal: Unknown executable stylish-haskell in package
stylsh-hskll-0.11.0.0-646a8e66

So I guess we might want to

  1. Make the target selector syntax a bit more ergonomic. I think this has to do with how we represent extra-packages as NamedPackage. They don't carry component info like the other local package types do and can't give good hints for the target selector resolution.

  2. Figure our why the cabal run code can't find the exe in the package plan (see matchingPackagesByUnitId)

michaelpj commented 4 years ago

Yes, I tried extra-packages, and it's close to what I want, but @phadej indicated that it's somewhat unsupported, so I'm abandoning that. I also hit the issues you mentioned with accessing the executable component properly, but I assumed that was down to extra-packages being janky.

However, I've realised that I can do what I want by wrapping cabal v2-install:

I'm going to try out a little tool based around this idea.

alexbiehl commented 4 years ago

I had a quick look and the selector problems seem to be solved in HEAD and I whipped up a quick patch in #6972 to make extra-packages work correctly!

Martinsos commented 2 years ago

I was just asking on reddit about the same feature for Haskell projects, basically what npm offers via package.json and its dev-dependencies field, so I am happy to have found this issue here!

@michaelpj what is your current solution right now?

For the Stack project I am working on, I wanted to have this experience with ormolu, stan and hlint. Ideally with hls also but I left that one out for now as I am not sure how to approach that. What I ended up doing (with help from @philderbeast - thanks a lot for the idea and initial implementation!) was defining individual stack-ormolu.yaml, stack-stan.yaml and stack-hlint.yaml files that all specify stack-snapshot.yaml as resolver -> package.yaml also specifies it as a resolver, so that way I ensure all of the tools are using the same resolver as my project. This is most important for stan, but it also gives me reproducibility for hlint and ormolu.

Then, to install any of these tools, e.g. ormolu, I use

stack install ormolu --stack-yaml=stack-ormolu.yaml

which installs it into project-local .bin directory, since that is specified in stack-ormolu.yaml via local-bin-path option.

Now I can use ormolu as .bin/ormolu.

From what I understood, similar thing can be achieved for cabal project by running

cabal install stan -w ghc-8.10.7 --installdir=.bin --install-method=copy --overwrite-policy=always

To make these tools even easier to install and use, I also implemented a run script that kind of plays the role of scripts in npm run/package.json -> it contains a list of commands that use locally installed tools (ormolu, ...) in opaque manner (they also install them if they are not installed yet). So using this script, you can just do ./run ormolu and it will install ormolu if needed and then also run it.

How this looks right now:

Does this count as a kind of an external prototype?

I understand that it takes some time for these tools to build per-project, but I don't mind as I find it very valuable to have this super simple process of just calling ./run stan or ./run hlint and it all works out of the box, for any developer on any machine, without them having to know anything about these tools or how to set them up for this specific project. This also makes it very easy for me to have the same situation both locally and on CI.

What I find really valuable with npm/package.json is this ability to define simple commands that run in the environment that has dev dependencies loaded into it. There is also npx tool that comes with npm that runs allowing dev dependencies directly, for example npx stan would run stan if it was installed as a dev dependency.

alexbiehl commented 2 years ago

@Martinsos Have you tried using extra-packages with a recent cabal release? I think it does exactly what you are after:

Try putting 

extra-packages: ormolu, stan, haskell-language-server

into your cabal.project file. You can then use cabal run ormolu to run the tools.

Martinsos commented 2 years ago

@Martinsos Have you tried using extra-packages with a recent cabal release? I think it does exactly what you are after:

Try putting

extra-packages: ormolu, stan, haskell-language-server

into your cabal.project file. You can then use cabal run ormolu to run the tools.

Hm this indeed looks like a solution (for cabal), thank you!!

I haven't found much documentation on this, so couple of questions:

alexbiehl commented 2 years ago

I haven't found much documentation on this

Right, the feature only started to work since I put a small patch in last year. And as such documentation is still quite a bit lacking.

Is there a way to ensure these tools are built with the same version of compiler as is used for the project? Will setting with-compiler in cabal.project ensure that?

Yes, with-compiler will do the right thing.

Is there a way to get it to build newer version of ormolu, since I know that it doesn't matter that stan is using older deps than ormolu?

Great point! The cabal solver does have a notion of solving exes from packages differently. We should make use of that for extra-packages. But for the time being I think it's not going to work unfortunately.

Martinsos commented 2 years ago

I haven't found much documentation on this

Right, the feature only started to work since I put a small patch in last year. And as such documentation is still quite a bit lacking.

Is there a way to ensure these tools are built with the same version of compiler as is used for the project? Will setting with-compiler in cabal.project ensure that?

Yes, with-compiler will do the right thing.

Is there a way to get it to build newer version of ormolu, since I know that it doesn't matter that stan is using older deps than ormolu?

Great point! The cabal solver does have a notion of solving exes from packages differently. We should make use of that for extra-packages. But for the time being I think it's not going to work unfortunately.

Thanks for putting in the work!

with-compiler sounds great then -> I think the only thing missing is getting cabal solver to understand it is executables. But that is problem only when one of the executables is somewhat older I guess / unmaintained.

fgaz commented 2 years ago

Then I guess we can close this in favor of a new "independent goals in extra-packages" ticket. I'll do that in a few days if nobody objects.

gbaz commented 2 years ago

@fgaz feel like getting around to the above? :-)

Mikolaj commented 2 years ago

/me vehemently refuses to object

fgaz commented 2 years ago

Alright, here it is: #7865