Being able to build smaller containers

[blackheaven]

I'm trying to define some simple containers:

  containers.blue = {
    name = "blue";
    copyToRoot = pkgs.buildEnv {
      name = "image-root";
      paths = [ blue ];
      pathsToLink = [ "/bin" ];
    };
    entrypoint = [ "/env/bin/blue" ];
  };

blue being a haskell.nix derivation

It creates a first layer with my derivation (~420MiB) and adds a layer with many things (coreutils-full, bashInteractive, su, etc.) which weight 15 GiB.

We should be able to passe the final derivation, or to disable extra layers.

[domenkozar]

Fully agreed, someone needs to look into container.nix and expose a knob to disable the layering.

[therealpxc]

It creates a first layer with my derivation (~420MiB) and adds a layer with many things (coreutils-full, bashInteractive, su, etc.) which weight 15 GiB.

15 GiB

I don't think those packages you name can be the culprit. They have really small closures! Together they're probably only like 100 MiB. Something weirder is going on.

[domenkozar]

https://github.com/cachix/devenv/pull/1375

[ppenguin]

I was having the same issue when trying to build a production container which includes a simple binary produced by buildGoModule. (15GB still seems like there's something else wrong too though, maybe you can get "closer to reasonable" when doing like below with the isBuilding logic)

My reference use case:

1. existing Makefile powered build of a go binary including private go module dependencies
2. desire to use devenv as a "one stop shop" to replace my own hacks and avoid implementing extending these hacks with "production image" functionality.

Immediate problems not (yet) solved with devenv:

1. Can't add binary artifacts due to a bug and the fact that you'd have to stage them (a la flake) for them to be found, which I want to avoid (no binary artifacts in git!)
2. If I use --impure, I'd expect to be able to add local relative paths anyway to e.g. copyToRoot, but it doesn't have any effect (i.e. I get a "path doesn't exist in Git repository" error)
3. Adding the artifact as a nix package is unattractive:
   1. forces me to make a nix package for my artifact
   2. which is non-trivial if you want to use buildGoModule (due to private dependency/vendoring/sandboxing nightmares => the only way I could make it work is to use go mod vendor but for this I have to add vendor to git as well)
   3. Doesn't let me use existing build processes
4. The resulting container image is "too big" and appears to contain lots of unnecessary stuff, which might be a consequence of using nix2container?

To reduce the size of the container image I already did the following:

  packages = with pkgs; lib.optionals (!config.container.isBuilding) [gnumake go-swagger gopls nix-prefetch];
  languages.go.enable = !config.container.isBuilding;
  copyToRoot = [(pkgs.callPackage ./pdnsupdate.nix {})];

which produces a 434MB image for a binary of 16MB.

If I use the above remove container tooling mod, I get a marginally better result (381MB). Note that this already uses a full nix package which I was trying to avoid in the first place, i.e. the unattractive scenario (2) mentioned above. Since (among others) I saw gcc in the container's nix-store, I'd guess that this could be the (implicit) buildInputs of the buildGoModule closure? In other words, if we use a nix package in copyToRoot, we still would need a way to only copy the runtime deps?

I'm not at all familiar with nix2container, but my experience with dockerTools.buildLayeredImage is not too bad, and it seems to allow for quite good control over the resulting image. So maybe it's worth considering going forward? As an additional alternative or as a replacement for nix2container?

Also the container functionality forces using docker, i.e. doesn't give a choice to use podman instead, which would be nice.

[therealpxc]

I think the bigger issue is that we're using the usual

which produces a 434MB image for a binary of 16MB.

If I use the above remove container tooling mod, I get a marginally better result (381MB).

Yeah, that's in line with what I expected.

I think it's clear that the main issue is not the extra tools added to the containers by default, or even the particular tools used to generate the container images. The issue is that the containers are based on the shells, but more crucially that the shells have fat closures. The shells have big closures because they're normal devShells created with Nixpkgs mkShell, so they pull in the build-time dependencies of everything in config.packages. They also pull in a whole C compiler toolchain because we're using pkgs.stdenv in the mkShell invocations by default (you can get that out of there by configuring stdenv = stdenv.noCC).

There's a WIP PR in Nixpkgs that introduces a distinction between 'build shells' (shells automatically derived from a derivation and which assume you want build-time tools associated with included packages, like the devShells produced by mkShell that we're currently using) and 'development shells', where some tools might be included in a different sort of way.

Short of waiting for that PR, depending on it now, or otherwise hoping that it meets our needs, we could choose either to handle devenv's config.packages differently (i.e., so that it is not passed directly as the packages argument of pkgs.mkShell) or to supplement it with a similar argument that means, more or less, 'packages to include in the CLI environment for runtime use but whose build dependencies are not needed'. One workaround that works for dependencies like that is just to wrap them in a buildEnv call, and include that result, rather than the package directly, into our config.packages. I did that here in order to erroneously pull the nix CLI into the environment when someone includes a Nix SCA tool by enabling the Nix language.

One consideration is that we kind of depend on the 'build shell' behavior for our interfaces for including language-specific packages-- this is why, for instance, one can just put their environment's Python libs into config.packages instead of using python.withPackages. Given that adding language-specific deps with this kind of interface is something Nix newbies often (wrongly) assume will work, e.g. with environment.systemPackages, this might be an important UX/DX feature for devenv. So maybe we don't want to change how config.packages gets plugged into mkShell but we want to introduce another class of packages that gets included more 'lightly'? There's kind of a similar question/problem for defaulting to stdenv.noCC, which might be fine for many languages but not if someone needs to compile native extensions.

Should we do an experiment against that WIP Nixpkgs PR and see (a) if, using that instead of pkgs.mkShell, we can get one of these containers with a huge closure down to a reasonable size and (b) what kind of interface is natural for specifying dependencies of different 'kinds' (i.e., those where we do care about the tools required to build against them as source code, and those where we just want to include them as a CLI tool)?

I'd be down to try that if someone wants to provide a sample project where their containers/shells normally come out bigger than necessary.

[blackheaven]

@therealpxc Thanks for your comment, I actually didn't figure out that mkShell was used to generate the image.

If no one does, I can have a try this week-end.

[domenkozar]

See https://github.com/cachix/devenv/pull/1415

[blackheaven]

Actually, IIRC, mkShell is used by default, when no entrypoint is specified.

I have made some progress, so that, when !isDev copyToRoot is passed through, fixing container size.

[ppenguin]

I have made some progress, so that, when !isDev copyToRoot is passed through, fixing container size.

Fantastic, I hope I find some time to try that sometime next week! I assume we're still "plagued" by the "problem" that we can't copyToRoot "impure" artefacts as long as they're not staged to git though, which would need to be solved too for a better UX?