Allow different implementations between native/byte compilation modes

[Leonidas-from-XIV]

Desired Behavior

Some code is not portable between the bytecode and native compilation modes and they need different implementations. Unfortunately, it is currently not possible to define the same library twice with different mode settings nor does enabled_if support matching on the compilation mode.

Example

There's multiple ways to express what could solve this issue, but these are only attempts to solve it that I have tried but that have not worked:

(library
  (name foo)
  (mode byte))

(library
  (name foo)
  (mode native))

This could be problematic because foo is declared twice, so an alternative could be

(library
  (name foo_byte)
  (enabled_if (= %{mode} "byte")))

(library
  (name foo_native)
  (enabled_if (= %{mode} "native")))

And then the select instruction could be used to use dispatch on foo_byte/foo_native to create a foo library.

[nojb]

Can you provide a concrete example of this? Also, see #4639 for the case of C stubs.

[NathanReb]

CC @emillon @voodoos.

[NathanReb]

@nojb this is for https://github.com/NathanReb/ocaml-jit and MDX. Here what I'll call JIT is just an in process binary assembler emitter + loader, as opposed to the current native toplevel workflow which relies on calling external commands for this.

We'd like to easily make MDX use the JIT when available and fall back to the default toplevel when not, i.e. in bytecode (where it does not really make sense to use the JIT) or on platforms with unsupported architectures. The current JIT library's interface is the following:

val init_top : unit -> unit
(** Register the JIT so that the native toplevel uses it instead of
    the regular toolchain.
    Will do nothing if the JIT cannot be enabled, if e.g. the mode is bytecode
    or the architecture is not x86. *)

val can_enable_jit : bool
(** Signifies whether the JIT can be enabled based on the current compilation
    mode and architecture. *)

The idea is that on non x86 platforms, init_top is a no-op and can_enable_jit is false.

We'd like it to behave similarly in bytecode as it would allow us to write code that uses the JIT and remains compatible with all platforms, native and bytecode modes, etc...

[nojb]

Thanks @NathanReb. What's the problem with the obvious solution of writing a small script to "choose" the right implementation?

(rule (with-stdout-to bar.ml (run choose.exe {%dep:bar_byte.ml} {%dep:bar_native.ml})))

Personally I would favour using the existing facilities instead of extending the Dune language (which is already quite complex in parts), unless there is a strong argument for it.

The idea is that on non x86 platforms, init_top is a no-op and can_enable_jit is false.

This would not be covered by the "native"/"byte" modes, right?

[Leonidas-from-XIV]

This would not be covered by the "native"/"byte" modes, right?

This is already possible by enable_if evaluating architecture and enabling/disabling libraries, coupled with select, but given the mode is not available as a variable it can't be used for this purpose.

[nojb]

Defining dummy libraries using enable_if and using select to select the right implementation seems strictly more complicated than directly choosing the right implementation with the aid of a helper program; do you agree?

[Leonidas-from-XIV]

It is not a dummy library, since the stubs and other code are part of the library as there is no point to attempt to build the stubs on unsupported architectures, so it is indeed useful to not enable the library in that case.

This can't be solved by a helper program, because the helper program does not know in which mode dune is going to link the executable/library that is will be using the library, since that would happen after the program had any chance to run, potentially even after the library has been installed into a switch.

[Leonidas-from-XIV]

We experimented with attempting to override the .cma file of the "full" library with the "empty" dummy version, but dune (correctly) complained that there are two rules generating the same .cma file and it can't proceed.

[NathanReb]

@nojb I don't think your suggestion would work since dune would only try to generate the implementation once and then use it to build both bytecode and native versions.

[nojb]

@nojb I don't think your suggestion would work since dune would only try to generate the implementation once and then use it to build both bytecode and native versions.

Yes, I was confused. What is being asked for here is the exact analogue of #4639 but for OCaml implementations. We should make sure that whatever is decided here is compatible with that other PR.

[Leonidas-from-XIV]

Indeed, I believe if you could get enabled_if to switch on the compilation mode this sounds like a superset of #4639 since you could have different foreign stub stanzas in each library definition.

[emillon]

We experimented with attempting to override the .cma file of the "full" library with the "empty" dummy version, but dune (correctly) complained that there are two rules generating the same .cma file and it can't proceed.

It would be possible to somehow "assemble" a third hybrid library by copying the cma from a library and the cmxa from another one and convincing dune that this cma + cmxa are a library, a bit like we do for (foreign_archives). But I'm not sure that this is the right way to do it.

[rgrinberg]

@voodoos just finished doing this for stubs. I suppose we can do this for sources as well.