RFC: design for better GHC plugin support in cabal

[michaelpj]

This came out of some discussion on IRC. I started writing an issue, but on reflection I think there's a relatively plausible design sitting there, so I wrote it down.

Motivation

GHC compiler plugins are currently not terribly easy to use with cabal, and require some amount of manual fiddling. In addition, they are not specified as declaratively as they could be (i.e. cabal has no concept of a "plugin"), which prevents cabal from doing some things as well as it could do.

I think there are two broad kinds of plugin that we should be thinking about:

1. Package plugins

Package plugins are those which are required by the package itself. They usually have a load-bearing role, in that they are actually necessary to compile the package. An example is the record-dot-preprocessor plugin.

Today, these can be used by:

• Adding the plugin package to the component's build-depends stanza
• Passing the appropriate -fplugin GHC option

Package plugins as implemented today have a few weaknesses:

• They don't use -fplugin-package, since they are just specified as normal build-depends. This means they get unnecessarily linked into the final build product (https://github.com/haskell/cabal/issues/6169).
• They are solved for as part of the main build plan, even though (see above) they don't need to be linked into the final build product.
• If/when the upstream GHC support for a native package db for plugins is completed, cabal will not be able to use it (because it doesn't know they're plugins).
• Users have to specify the -fplugin options themselves, as well as any -fplugin-opt flags.

2. Global plugins

Global plugins are ones which users want to run on all packages that get built, regardless of whether the package itself says it requires any plugins. An example is the ghc-tags-plugin plugin. Often these are used to e.g. get additional compile-time information out of dependencies.

Today these are very difficult to use (https://github.com/haskell/cabal/issues/7685, https://github.com/haskell/cabal/issues/6307, probably more). Partly the issue is that doing this nicely requires both setting a GHC option for upstream packages (doable today in cabal.project), but also building the plugin package and ensuring it is present in the appropriate package db when building the upstream package (hard to do without hacks like specifying your own manually created plugin db).

Since these sorts of plugins are also frequently used just in development, it would be particularly nice if they could easily be specified via cabal.project.local (https://github.com/haskell/cabal/issues/6169#issuecomment-593062344).

Proposal

1. plugin-depends field for .cabal files

(Prior art: https://github.com/haskell/cabal/issues/2965, but I'm putting it a little differently, so I'll repeat.)

The goal here is to make the situation nicer for package plugins. To that end, we teach cabal a plugin-depends field for component stanzas for .cabal files. plugin-depends contains a list of package/module pairs.

Initially, it could start out as a simple wrapper for build-depends, and then we could progressively augment it to have the following features:

• Also imply the appropriate -fplugin flags. (This is why we need the module component! -fplugin needs the actual plugin module.)
• Be passed via -fplugin-package (in particular, I don't think this needs to be a prerequisite, contra https://github.com/haskell/cabal/issues/6307#issuecomment-607146612)
• Be included in an appropriate native package db when/if that exists
• Be solved for as part of an independent build plan for native dependencies (relies on the previous item)

This deliberately doesn't cover -fplugin-opt flags. It seems reasonable for users to add those themselves.

(I'm also disagreeing with my past self here, who thought that we should just have native-build-depends and have that include plugins. However, the idea of including the extra information about modules that's needed to automatically pass -fplugin flags seems appealing to me, and requires a plugin-specific field. You could do this with a combination of native-build-depends and active-plugins (the latter taking a list of modules); maybe that would be nicer.)

2. plugin-depends package field for cabal.project files

The goal here is to make the situation nicer for global plugins. To that end, we teach cabal a plugin-depends field for package stanzas in cabal.project files.

This field simply augments the plugin-depends field for the corresponding package. Plugins that truly want to be globally applied can then use package * as usual.

[phadej]

plugin-depends having package name and module names will be messy. What if package provides multiple plugins etc. Just keep them separate.

Sometimes plugins need to be applied only in a handful of modules. That's another reason why plugin-and-splices-depends and ghc-plugins should be separate.

plugin-depends is/could be generalized to https://github.com/ghc-proposals/ghc-proposals/pull/243 and/or https://github.com/ghc-proposals/ghc-proposals/pull/412, i.e. TH dependencies.

Dependencies needed for TH would come from the same "host" dependencies.

-- I hope this name is awful enough it won't stick
plugin-and-splices-depends: overloaded, record-dot-preprocessor, my-awesome-type-class-th

-- Naturals is the option passed to overloaded.  Token syntax can be used here.
ghc-plugins: Overloaded Naturals, RecordDotPreprocessor

-- overloaded needs to be dependent upon, because it provides both the plugin and the library stuff.
-- there is no good reason to separate these today.
build-depends: overloaded

Existence of ghc-plugins could imply that compiler is ghc (and/or ghcjs?) for solver. Maybe even plugin-and-splices-depends too, as there isn't other compilers with TH.

[Mikolaj]

If this could solve https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5965, that would be indeed high priority to implement.

@bgamari

[michaelpj]

If this could solve https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5965, that would be indeed high priority to implement.

I don't think it would do that?

[gbaz]

@michaelpj do you still support this? I saw this linked from a newer issue, and I think this is a cleaner approach we should just do. I'm not sure I understand the concerns oleg raised, and I'm not sure they're significant enough to warrant against this simpler approach.

If you think this simpler approach is still valid, and still worthwhile, I think we would accept a patch for it.

[gbaz]

Hrm -- actually, we would want plugin-depends to alias build-tool-depends or setup-depends maybe, not build-depends, right? Then we get "independent solving" for free.

Or er... not sure actually. build-depends is certainly the easiest first step.

[michaelpj]

I do think this is nice. I continue to think we probably want a specific plugin-depends field given the existence of -fplugin-package and so on. I'm fairly convinced by Oleg that we should just leave it as the package dependencies and handle the specification of "active plugins" separately or not at all (let the user set -fplugin themselves). So plugin-depends would just be the package dependency.

It would be nice to have some opinions from other knowledgeable people, perhaps @angerman ?

If you think this simpler approach is still valid, and still worthwhile, I think we would accept a patch for it.

This isn't going to get to the top of my list soon, I'm afraid...

Hrm -- actually, we would want plugin-depends to alias build-tool-depends or setup-depends maybe, not build-depends, right? Then we get "independent solving" for free.

I don't really know enough about the innards of cabal to say what would be the easiest implementation strategy :thinking:

[angerman]

The core of this is in my opinion that we end up with two separate package databases; that have distinctly different purposes. plugin-depends is decidedly not build-depends. It's a runtime dependency for the build process. These should go into a separate package-db. I am against having auto stuff for -fplugin, if someone want's to use a plugin, use ghc-options for this coupled with plugin-depends. If you depend on the plugin, and the plugin also depends on a build-time library, you'll have to add it to both build-depends and plugin-depends. This is IMO the right approach. plugin-depends is much closer in spirit to setup-depends.

In a hypothetical future where cabal properly understands cross compilers, this will become necessary anyway, as the setup-depends/plugin-depends will have host native code, and the build-depends target native code.

The alternative would be to eventually have ghc package databases be multi-target aware, and then lump it all together into one package-db, but that seems a lot more messy than just cleanly separating package-dbs for their respective purpose.

[michaelpj]

Right, so my proposal here is to at least add the syntax and then at such a time as GHC can actually handle multiple package dbs, we'll be ready to take advantage of that. And in the mean time there are things like -fplugin-package and separate solving that we can do.

[angerman]

Correct. However even if GHC can handle multi-arch package DB's, cabal won't be worse of by being able to deal with/having a concept of multiple package DB.

[phadej]

cabal won't be worse of by being able to deal with/having a concept of multiple package DB.

plugin-depends and build-depends cannot be 100% independent though. The users will ask to be able to express constraints that plugin library and runtime library have compatible versions (somehow).

This problem already exists for build-tool-depends https://github.com/haskell/cabal/issues/5105, and while there one could argue (I did) that tool and runtime library are somewhat independent, being able to express the compatibility won't hurt.

In plugin case the dependency separation is less water tight. The plugin should be able to lookup symbols in runtime library (names with uniques), so plugin de-facto depends on runtime library too; they are just no inter-stage final product linkage.

So this makes me think that not only we need plugin-depends (for plugin users), but also runtime-depends (for plugin writers. c.f. executable-depends)

EDIT: this is conceptually the same as having TH-only dependencies. Code used to generate quotations is essentially a plugin in disguise.

[angerman]

Don't we have constraints for that? I've absolutely no objective to constraints applying to both. But they need to still be solved independently even if the same set of constraints are applied.

If you want to force the same version, == A.B.C would do so?

[phadej]

@angerman constraints are project level concepts. You cannot put constraints into .cabal file.

The plugin package author should be able to restrict the runtime versions of a library (and also simply add the runtime dependency). That is a cross-stage dependency, which should be expressible in the .cabal file. We cannot expect users to conjure the correct constraints to put into their cabal.project code for stuff to work.

[gbaz]

I think oleg is right that we will want some plugins that can be constrained to the same versions as linked at runtime, and some that can be independent.

The current proposal, in its initial form, by "under the hood" augmenting build depends means they're always linked, and I think that's better than nothing, as a start?

[angerman]

I have no specific view how that is solved. All I'm raising is that we must not assume that we can use the identical package, as the buildtime and runtime package can be different architectures. I'd hate to see any such assumptions (of identical architectures) being hardcoded in cabal.

[BebeSparkelSparkel]

I am trying to write a plugin but am having the problem where the ghc package requires transformers-0.5.6.2 but the package I want the plugin to modify requires transformers-0.6.1.1 which makes basically all plugins unusable.

[ysangkok]

@BebeSparkelSparkel It's usually not very hard to make packages compatible with multiple versions of transformers. Then you won't have to override the version of transformers shipped with GHC. Or you could update GHC to 9.10.1 and get transformers-0.6.1.1: https://gitlab.haskell.org/ghc/ghc/-/wikis/commentary/libraries/version-history

[BebeSparkelSparkel]

@ysangkok OpenBSD doesn't have 9.10.1 packaged yet and after multiple tries I have not been able to ghc correctly (it instantly seg faults when run). I'm using a custom mtl with some enhancements waiting for pull that I need more than the plugin.

Thanks for the suggestions though

[ulysses4ever]

Hello @BebeSparkelSparkel and welcome to the Cabal bug tracker!

You write:

I am trying to write a plugin but am having the problem where the ghc package requires transformers-0.5.6.2 but the package I want the plugin to modify requires transformers-0.6.1.1 which makes basically all plugins unusable.

I'm afraid your issue has nothing to do with Cabal but rather with GHC itself. The way GHC plugins set up is that they use GHC API, which amounts to depending of the ghc library. The ghc library version has to match the version of your compiler. The particular ghc library version depends on a particular version of transformers. As a result, the plugin also will depend on that same version of transformers. There's no way to change that version of transformers without also changing the compiler (for example, updating the compiler to a newer release or, in extreme cases, building your own version of GHC). So, you're stuck with the transformers that goes with the ghc library that goes with your compiler.

All of this is purely GHC (+plugins) architecture. So, I believe, your request is off topic on this thread, which talks about better support for plugins in cabal packages.

[BebeSparkelSparkel]

@ulysses4ever Thank you for more details. Let me give you some more just so that we can be sure that it has nothing to do with cabal and/or belongs in its own issue.

I'm using ghc 9.2.7 and cabal 3.12.1.0

ghc 9.2.7 uses transformers 0.5.6.2 (in my case)

It's fine if the plugin is compiled with transformers 0.5.6.2

The package I am trying to compile must use transformers 0.6.1.1

For me to use the plugin I think I must add it to the packages build-depends. If that is the case, then cabal reports a dependency conflict. From there I deduced that this is a cabal issue since cabal is intermixing the dependencies of the plugin and package.

Cabal should not be intermixing the plugin and package dependencies since the plugin that I am using only modifies the source and does not inject its own compiled objects into the package.

Maybe I'm way off here but it seems to make sense to me at least.

[geekosaur]

2965, cited previously, describes the problem and a potential solution.

[ulysses4ever]

@BebeSparkelSparkel okay, maybe you're right that the prosed plugin-depends could solve your issue, indeed. I'm sorry for the confusion...