Test opam constraints of multi-package repositories separately

[craigfe]

OCaml-CI's current strategy for running tests of multi-package repositories is to first pin all of the packages and then install them all in the same switch and run dune runtest. This approach has two undesirable consequences:

• missing external dependencies may be ignored by OCaml-CI, if those dependencies are introduced by another package in the same repository (e.g. irmin-pack and irmin-git both depend on alcotest, but this dependency is missing from irmin-git.opam);

• missing internal dependencies will always be ignored by OCaml-CI, since those dependencies are always available in the test switch by default.

It would be nice to have some CI solution that still tests that the opam metadata corresponds to the true project dependencies, as otherwise this becomes an issue that is only discovered at the point of release (see https://github.com/mirage/repr/pull/12). This is the major reason that we still have Travis CI running (very slowly) on certain mirage/ repositories.

CC @avsm.

[talex5]

I think the solution to this is:

1. Use dune to generate the opam files.
2. Make sure dune generates them correctly.

Dune knows which libraries go in each package, and which other libraries they depend on. It also knows which opam package those libraries are from. It should therefore be able to warn you if you fail to declare a dependency on an internal or external library.

(dune already added a check for the related problem of depending on another package using its internal name: https://github.com/ocaml/dune/issues/3727)

[craigfe]

For dune to solve this issue completely, it would need to also consider the cases of cyclic dependencies (annotated with post) and the distinction between test dependencies and normal dependencies. It seems to me that these are ultimately Opam's problem.

Also note that Dune's support for opam file generation is quite limited at the moment, often requiring us to use the .template fallback. These limitations would need to be lifted in order for Dune to become the real solution here. (c.f. https://github.com/ocaml/dune/issues/3770, https://github.com/ocaml/dune/issues/3653, https://github.com/ocaml/dune/issues/3497.)

[talex5]

I've made a separate linter that should help with this: https://github.com/talex5/dune-opam-lint/ It's currently enabled for testing (in #285).

It works by checking that every entry in your dune file has the corresponding opam package listed in the package's opam file. It's not quite the same as installing each package separately, but it will catch some cases that doesn't (e.g. a transitive dependency that works today but may disappear later).

It doesn't check internal dependencies at the moment, but dune can do that itself if generating the opam files (strict_package_deps).