building esy packages with nix

[timbertson]

:wave: Hello, I wrote and maintain https://github.com/timbertson/opam2nix

esy looks good, but it also makes me a little sad because it was a lot of work to support opam, and now all the cool kids are using esy ;)

It sounds like you're clearly aware of nix, and that esy is idealogically compatible given the digest + store model, plus sandboxing. So maybe there's actually some easier mapping into nix than there is for opam. On the other hand, I'm still very confused about the npm compatibility (do you support non-ocaml dependencies? how does that even work?) and the fact that the majority of the codebase is JS (maybe that doesn't matter, it just seems surprising for an ocaml project).

By way of an overview, at a super high level opam2nix works like this:

• opam2nix traverses the entire opam-repository and generates .nix derivations for every version of every package. These are fairly minimal, they mostly just declare dependencies.
• when building a toplevel package, we load this set of derivations into an opam universe and run the real opam solver given the package's dependency specs (this sounds like what you do). We use that to build a selections document (also a .nix file), which just imports the relevant versions of every selected package from the solution
• when actually building each derivation referenced by the selections document, we delegate to opam as much as we can, by passing it the opam file and extracting the build instructions. We can't literally use the opam binary, but the ocaml library is rich enough that we don't have to replicate too much functionality.

Given that, do you have thoughts on integrating esy into this? Or does it make more sense to meld it into one of the various npm->nix projects? Or is it unique enough it'd have to be its own thing? I guess the codebase is mostly a mix of npm repository logic with opam build/solve logic, so it's unclear which approach would be easiest.

This is probably more of a discussion than an issue, feel free to direct me elsewhere if this is a bad place to carry on that discussion...

[lobre]

Hi, I'd love also to see this discussion moving (even if time has already flown since the above message).

The onivim2 editor (https://github.com/onivim/oni2) is built using esy and I am not sure there will be a clean way to build it on NixOS until this gets figured out :-/.

[ManasJayanth]

@lobre Esy is now completely buildable with opam. In future, I'm hoping esy can be built with just ocaml and dune (both of which can be packaged for any os distribution) making it easier to distribute ./configure && make && make install styled tarballs

[ManasJayanth]

Regarding the original post, esy supports building sources from offline tarballs. Once esy itself can be distributed as a Nix recipe, esy packages being ported to Nix could be matter of exporting sources from esy store into tarballs that could be distributed with Nix derivations with esy as a build tool

[lobre]

@xavierzwirtz mentioned to me previously that having esy built itself as a nix package might not be enough as it would still be difficult to use it as a build dependency for of other nix derivations (for purity concerns around dependencies I guess).

And by the way, there is already an attempt to package esy in nix here: https://github.com/NixOS/nixpkgs/pull/96234

@timbertson, do you think that esy having support building sources from offline tarballs would help here?

[ManasJayanth]

I see those cached esy tarballs as the first step.

Ideally, esy packages are to be the transformed into Nix to play well with the rest of the ecosystem. But if it takes long to get there, we could start with the tarballs.

[lobre]

That would make sense. When you say esy tarballs, are you referring to what is explained on this documentation page?

https://esy.sh/docs/en/offline.html

If so, another small question: are these tarballs just an archive made of the sources of each dependency? Or are there already some kind of transformation happening when executing esy install --cache-tarballs-path=./_esyinstall?

(I am trying to figure out how to generate these tarballs in a "nix" way maybe using fetchGit or fetchTarball)

[xavierzwirtz]

@lobre, I don't think you will be able to generate the tarballs in a nix way, since they are going to require that esy have access to the network and the local cache to generate them. The best you can do is export the tarballs, and then commit them to a git repo for later use. That does sound like a good first step though, getting esy building things inside of a Nix derivation will probably be two somewhat separate projects.

1. Providing esy the sources that it needs to do its building.
2. Actually doing the build. This can be done without changes to esy most likely, the trick will be setting up an environment inside the Nix derivation that esy can actually build in.

[timbertson]

For context, it's OK if esy requires access to the network and local cache at resolve time.

In opam2nix (and most similar projects), there's two steps:

1. impurely, and with full network access, generate a solution / plan (to be committed in git)
2. at build time (i.e. inside a nix sandbox, with no network access), use parts of this plan to perform the actual build of each package

Step 1: plan

For step 1, what we'd need from esy is essentially a mapping of every package to:

• its URL / git repo,
• a copy on disk (in the esy store)
• its dependencies

From this, we can build nix expressions to recreate the tarball (we use fetchurl / fetchgit, and run nix-hash <local-copy> to get the expected nix digest). And using the dependency information, we can construct a nix expression where each derivation references its dependencies.

For a sample of the result of this step in opam2nix, see https://github.com/timbertson/opam2nix/blob/dc3397a71b058ce84807fc10f3069c14942e5cc3/examples/dev-dependencies/nix/opam-selection.nix

Esy wouldn't need to spit this out natively, spitting out a JSON with just the url / path / deps would be enough to fairly trivially build this nix expression.

Step 2: build

In opam2nix, this is done by each individual nix derivation having the same build instructions: "opam2nix invoke build; opam2nix invoke install". Each package runs this same build action, but with different environment variables (it sets things like the name + version of the current package, its .opam file, and a mapping of each dependency name to its path (e.g. "lwt": "/nix/store/XXXXXXX-lwt").

Opam2nix at runtime loads this information from the environment and reuses some opam library code to figure out what actual build command to run given the contents of the .opam file and the locations of all dependencies.

Given esy already has build plans, I'm hopeful that step 2 might be possible not by sharing code with esy, but by simply generating a compatible build plan with the nix store paths instead of the esy store paths. But I don't know what wrinkles would come up if we went down that path.

[jakeisnt]

Has any further progress been made on this after the plan? I'm trying to build an Esy project with Nix and this is the latest work I've found - and it looks like esy.lock as built by the project with esy provides all of the necessary information to pull these derivations into Nix (see onivim2 for an example).

What's missing to produce esy2nix from here?

[timbertson]

What's missing to produce esy2nix from here?

Funny you should ask! It turns out, a very large amount of code 😉

This is super experimental, and not building it in ocaml was probably a huge mistake (but I have my reasons 🤷 ). But it can build oni2. Perhaps nothing else at this point...

[phaer]

@timbertson Your fetlock repository does not seem to be available anymore? Did you end up deciding against the approach outlined above?

[timbertson]

oops, it was still set to private (public now). It's definitely not something I'd support any time soon, but you're welcome to give it a try