feature request: better support local reproducible environments

[tsnobip]

A lot has been done already to support local reproducible environments with opam, local switch first, and then the use of .opam.locked along with the --locked option, but I think the current setup still has quirks and unexpected behaviors.

For example, opam install . --locked doesn't uninstall dependencies that are not listed in the lock files, unlike what would happen in similar conditions with package managers from other languages (I'm thinking about yarn/npm for js/node or pip in python).

Overall, I think having a clear and documented way to achieve something like syncing local switch dependencies with a lock file would be nice and coherent both with the purpose of local switches used with a lock file and with many modern package managers of major programming languages.

[rjbou]

Lock files purpose is to lock dependencies of given package, at a current switch installation state, in order to share working dependencies. It's a simple opam file with more constrained dependencies. Its goal is to integrate with an already existent switch.

If you want to share full switches, you can use opam switch export/import, it even have option for reproducibility, as --freeze and --full:

If --full is specified, it includes the metadata of all installed packages
If --freeze is specified, it freezes all vcs to their current commit.

[rjbou]

Does it resolve the issue?

[tsnobip]

This solution is quite interesting but I'm actually not sure it solves all the use cases, for example we were building docker base images of our project with just the dependencies, we were using

RUN opam install . --deps-only --locked --with-test

for this, but opam switch import doesn't seem to allow to import dependencies only. I tried copying only the switch export file and importing it without copying the source but I was getting fatal errors:

#15 0.877 Fatal error:
#15 0.877 Base64.Invalid_char

By the way, is there any convention when it comes to name the switch export file?

[rjbou]

What is you usecase exactly?

By the way, is there any convention when it comes to name the switch export file?

Not really, usually the extension .export i think.

[tsnobip]

What is you usecase exactly?

We have a monorepo with many libs and executables in it.

We've found ocaml/setup-ocaml action to be quite slow to build and not very efficient at caching, so we use multi-stage docker images to benefit from more aggressive caching. We first build a base docker image that only consists of the dependencies listed in the .opam file of the monorepo. The docker images of the different libs and executables of the monorepo are then based on this image. This make builds quite fast when the dependencies don't change.

Right now we use a lock file to make this build reproducible, using opam switch export/import looked like an even better alternative but unfortunately doesn't provide a way to import a switch with only dependencies.

I'm not exactly sure why our use-case seems to be not common in the ocaml world.

[kit-ty-kate]

(oops i forgot to click on the "Comment" button)

We discussed that in dev meeting today. So far our conclusion so far would be that we'd need to add some sort of option enabled at switch creation (modifiable afterwards), to formally link a local switch to the opam files in the directory. Such an option would make it so that anything not in the opam files or in the invariant would be removed

[avsm]

@tsnobip here's an example of a multistage Dockerfile that follows the workflow you describe: https://github.com/ocurrent/ocurrent-deployer/blob/master/Dockerfile

The opam deps here are built first, and then the main project in a separate layer, and it's incremental since only the *.opam files are copied from the source directory when building the dependencies.

Another full monorepo is Real World OCaml: https://github.com/realworldocaml/book/blob/master/Dockerfile

In this case, we just build the full site, and the dune cache can be preserved across docker invocations to make changes incremental and fast.

Hope this helps unblock your usecase without needing changes in opam in the short term.

[kit-ty-kate]

We've found ocaml/setup-ocaml action to be quite slow to build and not very efficient at caching, so we use multi-stage docker images to benefit from more aggressive caching. We first build a base docker image that only consists of the dependencies listed in the .opam file of the monorepo. The docker images of the different libs and executables of the monorepo are then based on this image. This make builds quite fast when the dependencies don't change.

Right now we use a lock file to make this build reproducible, using opam switch export/import looked like an even better alternative but unfortunately doesn't provide a way to import a switch with only dependencies.

If that's your only use-case for the lock file, why not using a fixed opam-repository instead? This is what we are using for ocaml-ci/opam-repo-ci/... For example:

RUN opam repository set-url default git+https://github.com/ocaml/opam-repository#<the-commit-commit-hash-you-want>

or

RUN opam init -n git+https://github.com/ocaml/opam-repository#<the-commit-commit-hash-you-want>

if you're starting from an uninitialized opam root.

This is what the ocaml/opam images do by default by the way

[tsnobip]

wow thanks a lot for all your answers, I'm going to experiment with those different solutions then :)