RFC: Multi-context library installation

anmonteiro commented 1 month ago

Context

"Multi-context" or "Universal" libraries refer to libraries sharing the same name, but enabled in different dune build contexts (through (enabled_if (=%{context_name} ...))). Support for this was added in ocaml/dune#10307.

ocaml/dune#10471 shows that these libraries cannot currently share the same public_name.

Problem statement

Multi-context libraries may be installed by running dune install and passing the --context flag.
ocaml/dune#10493 shows the limitation when not passing the --context flag.

Proposed solution

Build Melange libraries & melange.emit exclusively in their own context.
Introduce a (melange) field to (context ..) in dune-workspace and phase out (modes melange) from the dune configuration
Introduce a new (context ..) field in (library ..) that adheres to the ordered set language:
- :standard defines the default context where most libraries live today
- libraries can be added to multiple contexts with e.g. (context :standard melange other-context)
- (context melange) implies (modes melange)
- (context :standard) (modes melange) becomes forbidden
Allow libraries with the same public_name to coexist iff one lives exclusively in the default context and the other lives exclusively in the melange context
- This doesn't represent a problem for installation because melange artifacts are already installed in their own directory: <lib-root>/<lib-name-segments>/melange
- Installing sources may have to be changed to install Melange sources in the same melange directory mentioned above
If --context isn't passed to dune install, automatically install the Melange and default contexts in the same prefix

The following example illustrates how a Melange project could be configured in this proposal:

;; dune-project
(lang dune 3.17)

(using melange 0.2) ;; likely implies incrementing the Melange rules version

;; dune-workspace
(lang dune 3.17)

(context default)

;; explicit in this proposal, to be discussed whether it makes sense to
;; automatically define the context if `(using melange <version>)` is
;; present in `dune-project`
(context
 (default
  (name melange)
  (melange <options?>))) ;; <-- NEW

;; dune

(library
 (name foo)
 (public_name foo)
 (context :standard))

(library
 (name foo)
 (public_name foo) ;; same public_name as the native library
 (context melange)) ;; new field `context` in `(library)`

Alternatives Considered

generate 2 dune install commands in the <project>.opam file
- this was ruled out because dune should be opam-agnostic, and other tools (e.g. nix) wouldn't have a way of knowing whether they should run dune install twice
implicitly define a "melange" dune context if (using melange <version>) is present in the dune-project file
- this isn't ruled out as a future iteration, but it probably warrants further discussion
Installing Melange libraries into dedicated prefixes (e.g. a sysroot like cross-compilation)
- this would likely result in unnecessary churn for package managers; in the case of OPAM, cross-compilation setup is quite niche and rather difficult to setup
- Universal libraries and Melange code aren't exactly cross-compilation, based on the necessity to vary entire implementations based on the target, whereas x-compilation is assumed to compile the same codebase, but varying the (cross-)compiler used

anmonteiro commented 1 month ago

This would likely also solve https://github.com/ocaml/dune/issues/10362

rgrinberg commented 1 month ago

The main thing that I dislike about this proposal is that it mixes low level constructs such as contexts and high level constructs as melange settings. The way things were originally, it made sense to me because contexts were expanded to cover more use cases without adding melange specific features. Now it turns that you need melange specific features anyways. In that case, we could have just started with something custom made for melange and the end result wouldn't have some of the sharp edges of this proposal.

What could a custom solution could look like? Well I suppose you could just make libraries built with the melange mode completely have their own sources and compilation artifacts and add conditional source selection for melange (though this would not work for bytecode/native/jsoo modes).

jchavarri commented 1 month ago

If I understand correctly, the main addition in the proposal is the melange field for contexts, which would mark a specific context as "Melange context", so that installation in those cases can be more lenient.

However, telling Dune about Melange-specific compilation at the context level feels like the wrong place (maybe this is what @rgrinberg is also expressing) because inside the contexts, one can put a lot of things besides Melange libraries or emit stanzas. This friction can be noticed in rules like:

(context :standard) (modes melange) becomes forbidden

I assume a bunch of other things would become forbidden that are not mentioned. For example what happens if I enable some executable stanza only inside a "Melange context"? One would expect that the "context => mode" path (which is the direction I understand the proposal is following) would run into other similar friction like that.

I wonder how a proposed solution would look like if we went the "mode => context" path. Meaning, we leave modes as is, and we somehow infer the context based on it. We could do this transition progressively by adding a config in dune-project e.g. (implicit_melange_context), and then have all libraries and emit stanzas with modes melange be implicitly added to it. That way, there are no conflicts to handle because the source of truth remains at the modes level (which wouldn't need to be phased out).

This would likely also solve #10362

That issue is mostly the same idea that I described above.

anmonteiro commented 3 weeks ago

What could a custom solution could look like? Well I suppose you could just make libraries built with the melange mode completely have their own sources and compilation artifacts and add conditional source selection for melange

I've been thinking about this and it looks like the most straightforward solution. Additionally, this preserves the status quo of Melange code not needing to live in a separate context -- I've been a bit unhappy about the limitation that we've self-imposed for universal libraries related to multiple contexts.

(though this would not work for bytecode/native/jsoo modes).

I don't understand this part, though. What wouldn't work?

rgrinberg commented 3 weeks ago

I don't understand this part, though. What wouldn't work?

All of those modes share build artifacts. Bytecode and native share cmi's and bytecode and jsoo share cmo's.

anmonteiro commented 3 weeks ago

I added a new proposal in https://github.com/ocaml/dune/issues/10630 that would replace this one.

andreypopp commented 3 weeks ago

The main thing that I dislike about this proposal is that it mixes low level constructs such as contexts and high level constructs as melange settings.

@rgrinberg can you elaborate on why you think contexts are low level constructs?

The appealing of contexts is that they introduce a modality (e.g. tools operate in different contexts operate differently, e.g. LSPs) and are already exposed in dune's UI (and we have added a switcher to vscode extension as well). So we can control diagnostics from melange or ocamlopt we get when viewing a source file (which is built by both).

But so far, as you've noted, the extensions to contexts are all generally applicable to any contexts.

Now we have few things which needs to be tweaked specifically for melange, and we have two RFCs now:

a designated special context for melange and do tweaks there (e.g. only libs with (modes melange) are added to melange context and how libraries are installed)
a completely custom support for melange as proposed in a parallel RFC

With 2. we also seem to be losing the modality I've described above.

rgrinberg commented 1 week ago

Contexts were added to support cross compilation. It's not clear if they are well suited for anything else because they do not compose and have poor performance characteristics. Teaching all the tools to know about contexts sounds like a whole lot of work compared to just implementing the bespoke feature in dune and making it available to everyone automatically.

It made more sense when the premise was that the feature was going to work with contexts unmodified. But now that it's no longer the case, we're getting the worst of both worlds.

So we can control diagnostics from melange or ocamlopt we get when viewing a source file (which is built by both).

There are much better ways of controlling this than contexts

ocaml / dune