`opam monorepo` support

[RyanGibb]

This PR adds support for opam-monorepo.

To support building mirage unikernels for cross-compilation targets we need to build them with dune since mirage 4 (in the absence of ocaml/opam cross compilation support).

Dune supports cross compilation with 'build contexts' using a toolchain which will use a target compiler installed in <opam-switch>/default/<toolchain> (as opposed to the host compiler). But not all dependencies are build with the target compiler. Notably ppx requires building with the host compiler. This information in stored in dune files -- e.g. here. In order to statically know how to build dependencies, we would have to parse the dune files to try and find out (which is non-trivial). Potentially a better solution is to encode the dependency build contexts in opam files, but opam doesn't currently support cross-compilation.

Instead, opam-monorep is used to build mirage unikernels. This fetches all the dependencies (filtered with ?monorepo in the opam file) to a local directory duniverse. Dune then builds the whole project. Unfortunately we can't benefit from the dune cache in our nix derivation, so all the dependencies will be rebuilt on every nix build.

This is implemented in the opam-monorepo project. This PR adds support for this workflow in the for of the queryToScopeMonorepo function, along with the lower layer functions mkSrcsScope, deduplicateSrcs, and defsToSrcs. We refactor the source fetching logic into evaluator/default.nix to share it between mkSrc (a derivation fetching a package's source) and builder.

To see an example of this usage see https://github.com/RyanGibb/mirage-hello. A flake template is the works.

[balsoft]

Overall a very neat implementation, seems to be minimal and mostly agree with upstream. Amazing job! The comments are mostly nitpicking, feel free to ignore them if you see fit. Please squash to one (or maybe a couple) commits before I merge this.

[RyanGibb]

Thank you! The comments are very useful :-)

I'm actually working on a buildOpamMonorepo to pick up pin-depends (for example in mirage-www) so will convert this to a draft this and a few other tweaks are complete.

[RyanGibb]

So trying use opam monorepo lock has more problems than I anticipated. First and foremost I can't get it to use local repositories.

However I don't think we actually need this. It's nice to have, but a work around for:

The case when the newer version does build with dune but an older version (which is overlayed) doesn't.

Is just to downgrade opam-repository.

This isn't ideal, but I'm happy to merge this as-is if you are and create a separate PR for this customer solver logic. Alternatively, I can carry on using the fork and let you know when I have more success.

[RyanGibb]

An aside: another use case I found for opam monorepo is picking up or dependencies. A workaround for this is just manually adding them to the query.

[balsoft]

This isn't ideal, but I'm happy to merge this as-is if you are and create a separate PR for this customer solver logic

Yep, I think this is a good-enough solution as it is right now. opam monorepo lock can be added in a later PR.

[RyanGibb]

Great :-)

[balsoft]

Could you clean up the commit history then?

[RyanGibb]

Done!