search

tweag / clodl

Turn dynamically linked ELF binaries and libraries into self-contained closures.
BSD 3-Clause "New" or "Revised" License
164 stars 6 forks source link

feat: adds tools for hermetic nix installation #48

Closed filmil closed 2 months ago

filmil commented 3 months ago

Applies the device introduced at: https://github.com/filmil/bazel-nix-flakes to produce a hermetic nix installation when bazel is started.

Made some changes to the clodl runner and pinned bazel version and options to ensure this still compiles.

Try it as follows:

bazel run :clotestbin

or

bazel run :clotestbin-cc
filmil commented 3 months ago

I'm not sure if you care for this device, but in case someone is interested, here it is.

facundominguez commented 3 months ago

Hello @filmil, thanks for your contribution.

I think the feature would deserve consideration for inclusion in https://github.com/tweag/rules_nixpkgs/issues/75, rather than in setup by users of rules_nixpkgs.

What do you think?

filmil commented 3 months ago

Well, this is not really a collection of build rules, so it can not be placed into a rules package.

I am not sure of the best way to enable easy addition into repos that need it.

I just thought it would be maybe an interesting approach to publicize. Especially because it just reuses existing tools in a new way, and as all nix+bazel attempts I've seen prior to this one were based on an existing system /nix/store. This requires a somewhat onerous nix installation, which is a big commitment for someone who isn't quite at that point in their dev journey.

This approach makes a private nix store that is tied to a specific bazel cache directory. Which makes it easier to use out of the box, as no special nix installation needed. And may be distributed cache friendly. I was a bit surprised how easy it was to apply to a new repo (clodl) once I worked out the exact needed changes. But the way this is done (copying files over) is not very pleasant nor maintenance friendly.

My goal is to help bazelize the world. In the past I worked out https://github.com/filmil/bazel-rules-bid, which is a way to run a single bazel build step inside a docker container. This avoids needing to bazelize things that are hard or onerous to bazelize, such as python programs with deeply nested dependencies. The downside there is the need to use docker, as podman does not work because of the nested sandboxing, and the general lack of hermeticity using docker images - though we still have reproducibility. It's also a bit annoying to maintain docker build environments for each project separately.

Hermetic nix+bazel be even better if there was a way to install nix in a user-defined prefix without needing chroot.

filmil commented 2 months ago

I modified the approach to make it not require elevated privileges for chrooting. Based on existing solutions (in this case, nix-portable). The glue is the only contribution.

aherrmann commented 2 months ago

@filmil Thank you for your contribution! This does look very interesting! Also thanks for documenting your approach so well. I've dug through it to get a better understanding of it (raw observations attached below in case they might be useful to others).

As far as I can see the following questions are open at the moment:

where to contribute this work?

I agree with @facundominguez that this could fit very well into rules_nixpkgs. As far as I understand this addresses issues that are relevant to all rules_nixpkgs users, not just clodl users. In particular, this provides support to build a Nix + Bazel project without a global Nix installation, not just to generate a relocatable and distributable artifact from a (Nix + ) Bazel project.

cc @benradf

in what form can this be contributed?

As you point out this does not provide a set of rules. So, it's not immediately clear how to contribute it to rules_nixpkgs. Perhaps there are ways to turn some of this into rules of some kind. But, before we go there. Maybe a good first step would be to contribute this as an example that is checked on CI. This way it provides documentation on how to use rules_nixpkgs without a global Nix installation right in rules_nixpkgs, and CI verifies that this approach continues to work going forward and is not inadvertently broken.

is there anything that should maybe be changed?

Two main things stand out to me that I think should change before this can be contributed:

  1. The use of git submodules. Git submodules tend to complicate the development and deployment flow and make it hard to keep versions in sync. And GitHub generated HTTP provided source tarballs don't include git submodules, which is a problem in Bazel, where the main distribution method is http_archive or variants thereof. Ideally, all the relevant code would be committed directly into the target repository, or downloaded through standard Nix or Bazel download methods, e.g. http_archive.
  2. Currently this checks in a binary blob, namely nix-portable. That's a supply chain risk for downstream users. How should they know which upstream version the blob corresponds to, and if no one has tampered with it. If a binary artifact is necessary, then it should be fetched from upstream releases and checked against a checked in integrity hash.

raw observations

filmil commented 2 months ago

Thanks for taking the time to look through this.

I agree with your findings.

As far as I understand this addresses issues that are relevant to all rules_nixpkgs users, not just clodl users. In particular, this provides support to build a Nix + Bazel project without a global Nix installation,

That seems to be the case. And that was the original idea.

Maybe a good first step would be to contribute this as an example that is checked on CI.

There is one at https://github.com/filmil/bazel-nix-example now.

  1. The use of git submodules. Git submodules tend to complicate the development and deployment flow and make it hard to keep versions in sync.

I understand that this is not ideal. It might be possible to bootstrap the approach somehow.

The issue is that the //tools/bazel script isn't really part of the build process, but is rather part of the bazel startup protocol. So our options to fix it may be limited.

For example, adding a self-contained repository implementing this, then using Alex Eagle's trick (used to be at 1, now 404s :/ ) to run a script from it that adapts the "main" workspace correctly.

  1. Currently this checks in a binary blob, namely nix-portable.

This is true. It should be removed.

  > /home/runner/work/bazel-nix-flakes/bazel-nix-flakes/tools/x86_64-Linux/proot: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory

FWIW: proot is not a viable approach. While I was eventually able to patch proot build to produce a fully static binary by removing the dependency on python3, proot has issues with handling the bazel server process because it gets daemonized, and proot waits forever for it to exit.

filmil commented 2 months ago

I reworked the approach to address the issues. I moved most of the setup into a separate repository, started downloading nix-portable via http_file, and removed the need to use submodules. It is now available at https://github.com/filmil/bazel_local_nix/pull/3. Note that the integration test passes.

I also minimized the amount of scripting that ends up injected into the top-level repository. I hope this relative opaqueness is not too problematic, since the scripts come from an integrity-checked regular repository anyways. If you worry about supply chain attacks (and it's everyone's top of mind these days, I wonder why), you should be careful about updating the integrity checks. Which one should always do anyways.

I also added an integration test repository, which is currently "just" a nix-based hello world program, but which compiles and runs as one would expect.

There seems to be a bug with nix-portable, where it insists to be invoked from the root of the repository on install. Which might not be too onerous since it must be done once on installation, and once when new nix deps are introduced.

It is also not strictly necessary to tack this code to rules_nixpkgs. It can live its separate life, and can be only brought in when needed. But it remains an option.

aherrmann commented 2 months ago

Sorry for the delay and thank you for addressing the comments and updating the code!

I moved most of the setup into a separate repository, started downloading nix-portable via http_file, and removed the need to use submodules.

Nice! That looks very good.

The issue is that the //tools/bazel script isn't really part of the build process, but is rather part of the bazel startup protocol. So our options to fix it may be limited.

I guess another simple solution could be a similar approach to what buildozer does: Download and cache a release binary on the fly and check it against an expected digest within the Bazel wrapper. That way users that decide to go with this approach could check in their tools/bazel wrapper without having to check in a binary but also without having to run an install step at every use-site.

It is also not strictly necessary to tack this code to rules_nixpkgs. It can live its separate life, and can be only brought in when needed. But it remains an option.

True. The way this is looking it could be a standalone Bazel module or it could be a part of rules_nixpkgs. I'd be very open to the latter.

filmil commented 2 months ago

I guess another simple solution could be a similar approach to what buildozer does: Download and cache a release binary on the fly and check it against an expected digest within the Bazel wrapper. That way users that decide to go with this approach could check in their tools/bazel wrapper without having to check in a binary but also without having to run an install step at every use-site.

Not sure I understand this bit. There is no need to check a binary in anymore. Nor is there an issue with checking in the tools/bazel script.

True. The way this is looking it could be a standalone Bazel module or it could be a part of rules_nixpkgs. I'd be very open to the latter.

Thinking a bit, I'm not sure we should require people who don't need an ephemeral nix installation to have to the support in.

I'm OK if the contribution stays as-is, and would welcome the folks at https://github.com/tweag/rules_nixpkgs/issues/75 to try out the ephemeral nix support at: https://github.com/filmil/bazel_local_nix

aherrmann commented 2 months ago

Not sure I understand this bit.

Sorry, that wording was a bit convoluted. The main point was "without having to run an install step at every use-site". IIUC, right now, every user has to run the install step by invoking bazel run @bazel_local_nix//:install. With the bazelisk alike approach that wouldn't be the case.

I'm not sure we should require people who don't need an ephemeral nix installation to have to the support in.

The way I was imagining it, it would still be an opt-in choice for users. If they don't set up the tools/bazel wrapper, then they won't be using the ephemeral Nix installation.

filmil commented 2 months ago

IIUC, right now, every user has to run the install step by invoking bazel run @bazel_local_nix//:install.

No, this is not necessary. As soon as someone checks in the results of the installation, it is done. No additional quirks. Binaries are fetched via external repositories as usual. The bazelisk or buildifier approaches are not needed. (since the binary in the rule does not need to exist outside of a specific project scope, while bazelisk or buildifier both do.)

Maybe I misunderstood your point?

The way I was imagining it, it would still be an opt-in choice for users. If they don't set up the tools/bazel wrapper, then they won't be using the ephemeral Nix installation.

While yes, you could do that, note that it is unnecessary. The ephemeral nix installation in rules_local_nix is fully optional. One opts in by adding the stanzas to bring the rules in, just as one usually does. It is also additional complexity that doesn't need to be introduced everywhere.

That said, maybe you'd want to take over the maintenance of the rules, in case I slide under a bus or something. In which case feel free to put them wherever you think they fit.