Smaller closure for CI - Githubissues

aspiwack commented 3 years ago

Hello.

I've just started using opam2nix, which I've deployed it on a CI to simplify caching. I noticed that there is a fairly non-trivial closure. Altogether, my dependencies take 1.5–2min to download (see e.g. this build).

While it's nothing to scoff at, I believe it could be brought down quite significantly. I noticed for instance that the step I gave downloads a bunch of Ocaml 4.10 dependencies (including the compiler itself). These are, I assume, build dependencies for opam2nix (my own project uses Ocaml 4.11). I'm not entirely sure how they came to be runtime dependencies, though see below.

For the sake of CI, really, I don't need opam2nix-the-tool. Only opam2nix.build. But I haven't found a way to use opam2nix.build without having the opam2nix binary in my closure (and, in fact, the library in all its .cm{i,t,x} glory). I suspect that the .cmt are the ones guilty of pulling all the compile dependencies as runtime dependencies.

I procure opam2nix by importing the repo like so

  opam2nix = import sources.opam2nix {};

The repo is cloned (by niv) at https://github.com/timbertson/opam2nix/archive/baeffc1078309f9fdd4e0abd7e1a1f762e92bbe3.tar.gz.

And, to reiterate, it seems that using opam2nix.build suffices to pull the opam2nix executable and the cmt files.

I'd be quite satisfied of any solution that lets me use opam2nix.build without pulling executable and derivation.

An alternative that I can imagine would be only move the library to a separate derivation, while still having the opam2nix binary available immediately.

I have no idea how to achieve either of these possible solutions. Therefore I don't presume to know which is easier, or, really if any is feasible. But shorter CI times are a worthwhile goal in my eye.

timbertson commented 3 years ago

Thanks for bringing this up. I haven't spent much time thinking about it, but I have a few initial thoughts:

opam2nix (the binary) must be a build time dependency, since it's used in buildPhase / installPhase of each derivation that it generates. This is because we load the opam file at build time to figure out what commands to run.

I don't know enough about ocaml outputs to understand the importance of the .cm{i,t,x} and which bits are necessary. I agree it shouldn't logically need all the ocaml libraries in its closure. Perhaps it's worth trying a fixupPhase to remove these? Maybe there's already something in nixpkgs?

It would be nice to have opam2nix build with your ocaml version. You can do this, by specifying ocamlPackagesOverride when you import opam2nix (but not via opam2nix.build, it's too late at that point). It's a bit odd and undocumented though. See https://github.com/timbertson/opam2nix/blob/baeffc1078309f9fdd4e0abd7e1a1f762e92bbe3/nix/default.nix#L2

I considered always doing this, but there are some times you wouldn't want it - you might want to use opam2nix with an ocaml that it doesn't compile under. Or you might be building packages against 5 different ocaml versions, and it's wasteful to build 5 opam2nix instances for that. But mainly it's just a bit messy to implement ;)

aspiwack commented 3 years ago

Aha! A piece of the puzzle falls into place. Fair enough, opam2nix is a build dependency of my project because my project is part of the selection.nix file. In an ideal word, I would manage to drop it from my dev environment. But, I say, fair enough.

Overriding the Ocaml package is a good first step. But it is but an ersatz of a more ideal solution, as we would still be depending on build dependencies of opam2nix which I don't need to run it.

Now, my knowledge of the Ocaml internals is pretty rusty, to say the least. However, I'm pretty sure that all these .cm{i,t,x} files are not necessary for the executable to run (in fact, I seem to remember that Ocaml programs are statically linked and fully relocatable). They are useful, on the other hand, when building an other Ocaml thing against opam2nix as a library.

.cmi files contain type information, they are used by the typechecker when you refer to the corresponding module
.cmt files contain documentation (doc strings, symbol locations). They are used by the likes of ocamldoc, odoc, merlin and ocaml-lsp. When built with Nix they appear to contain references to Nix store path (as revealed by calling strings on them). These are probably links to definitions of the types appearing in the interface or something like this. (I'm quite fuzzy on the details because my serious Ocaml time is largely pre-cmt)
.cmx contain unlinked compiled code. They are eventually stringed together into an executable by the linker. Statically, if I remember correctly.

They are all stored in the site-lib directory, which is where findlib looks for libraries.

So yes, it should be possible to drop them, if I'm to be trusted (which I am definitely not). Maybe by making a derivation which copies out the bin directory to a new store path? You'd need to make sure that the runtime dependencies are indeed propagated I don't know if it happens automatically.

$ strings /nix/store/z5ji7f8dp58ahyd084k0vi1ci5j
wwwl4-ocaml4.10.0-opam2nix-1.1.0/bin/opam2nix  
#! /nix/store/k8p54jg8ipvnfz435mayf5bnqhw4qqap-bash-4.4-p23/bin/bash -e
export PATH='/nix/store/axxnxbsppbprm07m677s83p3xafilfra-ocaml4.10.0-opam-installer-dev/bin:/nix/store/na7767q23fvcl64xmkj7q5hvnn4anmfq-nix-2.3.8/bin:/nix/store/fayjl0glacc00mk037nzvs0mnjysnz5v-git-2.28.0/bin'${PATH:+':'}$PATH
exec -a "$0" "/nix/store/z5ji7f8dp58ahyd084k0vi1ci5jwwwl4-ocaml4.10.0-opam2nix-1.1.0/bin/.opam2nix-wrapped"  "$@"

$ strings /nix/store/z5ji7f8dp58ahyd084k0vi1ci5jwwwl4-ocaml4.10.0-opam2nix-1.1.0/bin/.opam2nix-wrapped | grep "^/nix"
/nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib/ld-linux-x86-64.so.2
/nix/store/9gphp6kzv36z97qq7dim43gi958sz4k5-curl-7.72.0/lib:/nix/store/35wqyfdvvpd83zwalnp7g93wv64kz7my-libev-4.33/lib:/nix/store/bdf8iipzya03h2amgfncqpclf6bmy3a1-glibc-2.32/lib

timbertson / opam2nix

Smaller closure for CI #52