Steps to reduce overhead when generating Docker images

[DMills27]

During a conversation with @olirice last week, he mentioned that the generation of Docker images results in a significant amount of overhead. I noticed while perusing the codebase that dockerTools.buildImage is used in the flake.nix file and I suspect that this is what's generating the unnecessary overhead. My reason for saying this stems from the fact that buildImage insists on writing the entire image, with all its layers, to the Nix store (which in turn consumes unnecessary disk space since individual layers may contain information already present in the Nix store) and this is then compressed into a tarball where any change in a layer leads to a new Docker image being generated, in the form about another tarball, in the Nix store. To make matters worse, a lot of unnecessary duplication can occur between the Nix binary cache and Docker registry tarballs, since the heuristic used for layering prioritizes popular packages being grouped together.

There are a handful of ways in which this issue can be addressed. The most straightforward of which is to replace the dockerTools.buildImage function with the nix2container.buildImage function from the nix2container package which was reified from the contents in this blog post (see here for how it's used and the pitfalls to avoid when using it).

Alternatively, if greater control and further optimization is later required one can opt for one of two choices:

• Using a NixOS container via systemd-nspawn to generate a lightweight container. The beauty of this approach is that since systemd-nspawn is available on most Debian systems, by default, it doesn't lock one into using NixOS in and of itself. There are, however, a number of benefits to this approach if NixOS is chosen as the operating system under consideration, such as addressing https://github.com/supabase/nix-postgres/issues/13 in a more seamless manner, and they are detailed here. This might be useful to know as nix-postgres matures.
• Using nix-snapshotter. I've only recently come across this and it seems fairly new but at the same time promising since it claims to have advantages over nix2container in the README. Again, the upshot of this is that it doesn't lock one into the Nix ecosystem while leaving one with a lot of interoperability with it and more familiar tools. However, containerd, the container runtime it uses, seems completely new and perhaps one should see how it matures first.

[steve-chavez]

TIL many things! Looking forward to seeing nix2container integration.

[samrose]

closed with #45