Expose flags though files and enable the use of -runtime-variant

[TheLortex]

For mirage/mirage#969

Instead of pkg-config, one can use the following files to get compilation flags:

• ocaml-freestanding/libs
• ocaml-freestanding/cflags

For example with dune this allows to write %{lib:ocaml-freestanding:cflags} to get the flags instead of having a script invoking pkg-config ocaml-freestanding --cflags.

Moreover libasmrun.a is symlinked in the ocaml directory in order to be able to use ocamlopt's -runtime-variant option.

[mato]

@TheLortex This change looks reasonable in isolation, as I understand it your changes are orthogonal to the current OPAM/pkg-config based setup?

However, what I'm missing is the big picture/long-term overview of what you're trying to achieve, especially in relation to upstream (solo5-bindings-*). Is the plan to dune-ify everything, getting rid of the dependency on pkg-config along the way?

Also, what is the role of opam in this "new order", given https://github.com/mirage/ocaml-freestanding/pull/50#discussion_r257231892 by @emillon?

[TheLortex]

@TheLortex This change looks reasonable in isolation, as I understand it your changes are orthogonal to the current OPAM/pkg-config based setup?

These changes are orthogonal to the current behavior. It enables a new way of gathering flags that doesn't use pkg-config and that makes it easier to build downstream libraries with dune.

However, what I'm missing is the big picture/long-term overview of what you're trying to achieve, especially in relation to upstream (solo5-bindings-*). Is the plan to dune-ify everything, getting rid of the dependency on pkg-config along the way?

• The main plan is to dune-ify downstream libraries in order to be able to build an unikernel with dune instead of ocamlbuild (see mirage/mirage#969). One goal for these packages is to have as less external dependencies (pkg-config,opam) as possible and that's why I designed this change.
• Dune-ify everything is not on my roadmap as it requires extensive work for low-level packages such as solo5 bindings.

Also, what is the role of opam in this "new order", given #50 (comment) by @emillon?

Therefore opam is still relevant, but the goal is to depend on it as less as possible.

[mato]

These changes are orthogonal to the current behavior. It enables a new way of gathering flags that doesn't use pkg-config and that makes it easier to build downstream libraries with dune.

Right, but the new way is dependent on all downstreams using dune for building, correct? Also, pkg-config will not be gone entirely until it's also removed upstream (from Solo5).

So, effectively we're stuck with both mechanisms (pkg-config and this new mechanism) until some future time that everything is dune-ified?

Dune-ify everything is not on my roadmap as it requires extensive work for low-level packages such as solo5 bindings.

I'm the author and maintainer of Solo5, so I might be able to help with that, but am trying to understand the wider plan.

Is there some "Dune manual" or other document, or examples that I can look at to understand better what the final result would look like?

[mato]

(merging this as the discussion is not a blocker for that)

[emillon]

Hi @mato,

The ultimate goal here is to consolidate the build strategy for unikernels: make it possible to build unikernels themselves using dune, port the mirage libraries to dune, and make the different tricks used in mirage first class features. In particular, replacing the following current practices is in scope:

• "the linking trick" (building against an interface and only selecting the implementation when linking the final binary)
• building a library with several versions of stubs
• storing linking options in META files
• using pkg-config to build C stubs

@TheLortex made a list of different points that are missing, as well as a plan in mirage/mirage#969.

For this PR, the idea is to expose an alternative interface to packages that depend on ocaml-freestanding, so that these do not have to use pkg-config directly. This indeed requires a transition period where both a .pc file and these sexp files are installed at the same time, but after this transition is over (when Mirage 4 is released, more or less), I think that we'll be able to remove the .pc files from the opam installation, but this shouldn't be incompatible with solo5 using pkg-config for its own dependencies.

I hope that clarifies a bit the situation, let me (or @TheLortex) know if something could be clearer :)

[mato]

@emillon:

This sounds good, and thank you for the explanation.

A note about the use of pkg-config in Solo5: The reason it is used here is because this mechanism was originally used to pass around CFLAGS / LDFLAGS / library dependencies by mirage-xen, so we copied it for Solo5. In the future we may replace it with something else, however whatever mechanism is used (for example, presenting a proper cross toolchain) should not itself depend on dune or OCaml build tools.

Of course, downstream of the Solo5 packages, i.e. ocaml-freestanding and its dependencies are fine to transition to a dune-based mechanism entirely.