Question: Is it possible to use nixpkgs_package to obtain transitive closure of all dependencies

[groodt]

Is your feature request related to a problem? Please describe. Is it possible to take an attribute from a nix repository and bundle the full transitive closure into a tarball?

Describe the solution you'd like N/A Describe alternatives you've considered N/A

Additional context Just wondering if it would be possible to use Nix for more than tools in the Bazel build. Can the files be actually shipped. For example, layering awscli into a container_image rule for example.

[aherrmann]

That's a good question. A good way to achieve this is to construct the bundle within Nix and then point nixpkgs_package at it.

For example, a Docker use-case is documented here. That uses dockerTools on the Nix side to generate a base image, imports it into Bazel using nixpkgs_package, and then forwards it to rules_docker. Does that cover the use-case you had in mind?

[groodt]

That does look interesting. Thanks! I'll take a closer look and hopefully it answers my question.

[mfarrugi]

Has anyone worked out a solution to this?

To prototype how this could work I grabbed the transitive closure of a set of packages (as provided by nix-store -q --requisites $pkgs ) and pass those to a @rules_docker container_layer to build a final image that works.

A little more generally, I'm wondering: how do people use @rules_nixpkgs without packaging support? Are people building a nix package that contains their bazel build? Perhaps running the bazel build side-by-side for interactive development, and then running a second (pure nix?) build for packaging?

[groodt]

Yes, the solution posted above works works well.

The approach I've taken is to treat nix as a pure package manager and bazel as a build tool.

In other words, nixpkgs provides an environment with bazel and other toolchains for the build (including base docker images).

Then the bazel build handles the application build and layers that on-top of the base images.

In that way, I ensure the tool in the docker image is the same tool used during the build.

[mfarrugi]

Is that approach not difficult to maintain? If you need a separate nix base image for each *_binary/_image packaged, you need to declare every build target's dependencies in both nix and bazel right (or add some package to the base image that is non-obviously connected to the dependent library)?

Ideally I am thinking that bazel can sniff through a *_binary's runfiles to find the nix store paths referenced and then create the nixpkgs image with a layer per-package. Does that sound like it would remove some boilerplate for you?

On Mon, Oct 25, 2021 at 6:05 PM Greg Roodt @.***> wrote:

Yes, the solution posted above works https://github.com/tweag/rules_nixpkgs/issues/149#issuecomment-738063342 works well.

The approach I've taken is to treat nix as a pure package manager and bazel as a build tool.

In other words, nixpkgs provides an environment with bazel and other toolchains for the build (including base docker images).

Then the bazel build handles the application build and layers that on-top of the base images.

In that way, I ensure the tool in the docker image is the same tool used during the build.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tweag/rules_nixpkgs/issues/149#issuecomment-951368260, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAROG6CU4XR55DM5UBDUY3TUIXICRANCNFSM4UKKHFPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[groodt]

Is that approach not difficult to maintain? If you need a separate nix base image for each *_binary/_image packaged, you need to declare every build target's dependencies in both nix and bazel right (or add some package to the base image that is non-obviously connected to the dependent library)?

Not really. Typically there are a handful of languages in the bazel monorepo, so there would be base images with the JRE, base image with python3. These base images are then pulled into the bazel WORKSPACE and the application-specific artifacts are layered on top of the handful of standard base images. i.e. uberjars in the case of Java, the py_binary in the case of Python, compiled haskell code in the linked example.

So I must be clear, nix isn't building custom base images for each runnable target, I leave that to Bazel. Nix is only supplying a handful of language specific runtime base images. I leave bazel to handle the application build and build of third-party dependencies that are used in the application. So in the case of Java jars, these are pulled using bazel rules, not nix. The same for python deps, they are not pulled in via nix. It is certainly possible though to have nix build a "vendored" set of artifacts if you do wish to, but I'm not currently doing that.

[mfarrugi]

I see, thanks for the explanation. I am looking at using nix for cpp and python package management, so that's the main difference in our experience.

On Mon, Oct 25, 2021 at 6:46 PM Greg Roodt @.***> wrote:

Is that approach not difficult to maintain? If you need a separate nix base image for each *_binary/_image packaged, you need to declare every build target's dependencies in both nix and bazel right (or add some package to the base image that is non-obviously connected to the dependent library)?

Not really. Typically there are a handful of languages in the bazel monorepo, so there would be base images with the JRE, base image with python3. These base images are then pulled into the bazel WORKSPACE and the application-specific artifacts are layered on top of the handful of standard base images. i.e. uberjars in the case of Java, the py_binary in the case of Python, compiled haskell code in the linked example.

So I must be clear, nix isn't building custom base images for each runnable target, I leave that to Bazel. Nix is only supplying a handful of language specific runtime base images. I leave bazel to handle the application build and build of third-party dependencies that are used in the application. So in the case of Java jars, these are pulled using bazel rules, not nix. The same for python deps, they are not pulled in via nix. It is certainly possible though to have nix build a "vendored" set of artifacts if you do wish to, but I'm not currently doing that.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tweag/rules_nixpkgs/issues/149#issuecomment-951390977, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAROG6C5FJW7JBWOUWC76XDUIXM3RANCNFSM4UKKHFPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[aherrmann]

That does look interesting. Thanks! I'll take a closer look and hopefully it answers my question.

@groodt Did that answer your question? I'll close for now. Feel free to reopen if it didn't.