Lock directories are currently specialized to the system where they were generated

[gridbugs]

One of the goals of the Tarides build-system team for this quarter is getting dune package management into a state where it can realistically be used instead of opam. We identified that the current DX around lock directories will likely hinder this in practice.

Specifically, there is a convention in other programming language ecosystems that lockfiles should be checked into version control, and the same lockfile be used by all developers of a project, possibly spanning different operating systems and architectures. Thus it's not unreasonable to expect users of dune package management to check lock directories into version control. Dune lock directories are often highly specialized to the system where they were generated and can't be used on different OSes or architectures. One reason for this is that the ocaml compiler package has dependencies on packages named host-arch-XXX and host-system-YYY that are specialized for the current architecture and OS, and dune's solver only produces a package solution compatible with system where it is running. Consequently, if a project checks in their dune lock directory, it will likely lead to unexpected results for other contributors whose computers are different from the computer where the lock directory was generated.

There are technical justifications for why dune's solver works this way related to how dependencies are specified in opam packages, but these are out of scope for this issue. Suffice it to say that changing dune to produce lock directories that can be guaranteed to work on systems other than the one where they were generated is not something that we plan to work on in the near future, and we'd like to start encouraging wider use of dune package management before this work is done.

In the short term, I believe that one way to save users from the headache of checked-in lock directories that are incompatible with their OS/architecture is to change the default name of the lock directory from dune.lock to .dune.lock or _dune.lock, and for projects initialized with dune init, to add an entry to the gitignore file to hide the lock directory. Possibly also print a message when generating lockdirs that they aren't currently intended to be checked into version control. Hopefully this will encourage users to use lock directories the same way as opam switches are currently used, in that they allow isolated package sandboxes per project, with the benefit over opam that users only need to install a single tool (dune) and that they will be faster to create (as no package's will be built when generating them).

Longer term we can look at generating lockdirs that can be checked-in and used across different OS/architecture combinations, but in their current state lockdirs are still a valuable tool, and we don't want to delay releasing dune package management while we make the changes necessary to enable this workflow.

[Leonidas-from-XIV]

Yes, I think we had a discussion about this that .dune.lock is the way to go. A similar thing goes for the lock files of the dev-tools where even if they were os-independent we probably don't want to incentivize people to check them in because the exact dependency cone is not relevant for the dev-tool as long as the user gets the right dev-tool in the end.

[rgrinberg]

Possibly also print a message when generating lockdirs that they aren't currently intended to be checked into version control.

Without checking them into version control, it's basically impossible to produce a reproducible (even if platform specific) build plan. Maybe that's not a goal for everyone, but it's certainly useful for some people. So I think this message could possibly misguide users that there are no use cases for checking in their build plan.

You could also just save the temporary build plan in _build. It's not technically a source directory, but I think we already use it for other random things.

[rgrinberg]

By the way, we could always allow for a mode that locks the "inputs" rather than "outputs". I recall that this was David's original idea to just state that the commit hash of the opam repo would be your lock file. We already save that commit hash, so all we'd need to do is to add the pinned package url's. This mode has many downsides so it's not a replacement for the current lock directory, but at least it maintains reproducible builds unlike this proposal.

[gridbugs]

Ok so as a conservative first step, let's just rename the lockdir to .dune.lock and leave it up to users whether or not they decide to check it in. When we document all this stuff we can describe the pros and cons of checking in lockdirs.

[jberdine]

Maybe I'm overlooking something, but what is the utility of a lock file to users if it cannot be checked in and shared with other developers to recreate the same development context? Phrased differently, if the lock file is specific to the system that generated it, then isn't it a cached result of a build step (using the term build step very roughly)? It seems like it has a similar status to _build/**/*.dune-package files, and putting it in _build would signify that. Just giving it a name starting with . isn't enough I think, as many configuration files have such names and should be checked in.

[rgrinberg]

Why do you think it cannot be checked in? Some are saying that it shouldn't be checked in because it is not portable enough. It is certainly not a universal concern though.

[jberdine]

My thinking is that if it isn't universally robust enough to recommend checking it in, then I don't want to check it in and tell potential contributors to a project that the way to build is to use the lock file. Maybe I'm misunderstanding, but I'm thinking by analogy with e.g. opam lock files where I can tell contributors not to use the opam solver, etc., but to use exactly the listed package versions.

[rgrinberg]

To make a build plan "portable", we have to clear two hurdles.

1. We need to select package sources according to the platform
2. We need to execute the package build scripts with the platform specific variables

The first hurdle isn't cleared by opam's lock files nor our own lock directory.

The second hurdle is cleared opam's lock files, but isn't a completely solved problem in our own lock directories. There are semantics of opam's filters and variable evaluation that are to difficult replicate in a way that respects the semantics of the dune language. It's not an insurmountable difference, so I am hopeful that we'll solve it one day. However, many packages are don't really have build scripts that are conditional on the platform - unfortunately, some of those packages are important.

My own opinion is that 1. is essential and therefore neither solution is portable in practice. To actually gain portability in any meaningful way, one has to define the set of platform configurations that are supported, generate lock directories for them, and most importantly, test that these configurations actually work. The generation step can be done any platform in dune, but dune has no way of helping you with the second step at the moment.

[jberdine]

I think the situation would be quite good if a small finite set of known good configurations could be maintained and checked in, where it was possible to reliably determine if a machine trying to perform a build matches one of the checked-in configs.

[rgrinberg]

Coincidentally, that is a workflow that we currently support. Although it requires a bit of config in the workspace file write a conditional to select the lock directory based on the platform. Mostly because we are still figuring out what is the good default here.

[jberdine]

Nice. Having config in the workspace file (and checking it in) is already where things seem to end up in order to have multiple build profiles simultaneously, even for a single opam switch. So no loss there.