Investigate using OverlayFS for nixos containers

[spacekitteh]

Now that OverlayFS supports SELinux, it is much more suitable for linux containers. Thus, it should be investigated.

(Sidenote: how do nixos-containers and LXC compare?)

[joachifm]

NixOS containers are systemd-nspawn+bind mounts. My impression is that overlayfs is commonly used to run container instances from a (shared) pristine OS runtime (to maximize sharing & having a sane way to do upgrades), but that's not too relevant for us, I think.

[fpletz]

I've some changes in my stash which bind-mount only the needed store paths into a container. I didn't push it yet because imperative containers are broken with this code and supporting both would require some refactoring.

It's remarkable though that I didn't notice big differences in performance while (unscientifically) running the container tests on my laptop. We also should check how many mounts are possible, both per namespace and globally in the kernel.

To protect the nix store of the host we only have two solutions: Either via bind mounts or by copying the system closure into a new nix store. Currently, nixos-containers even use the nix-daemon of the host which while being handy is a major security risk because it's running as root.

Does it make sense to support both approaches? For imperative containers, we would need support for copying the system closure anyway if we want to separate the store of the host and the container.

[Mic92]

@fpletz yes it makes sense to have both options: copy vs bind mounts - we did something to build rkt images: https://github.com/Mic92/nix2aci. How did you perform the mounts? My idea was to make an shared bind mount of an arbitrary directory (let's called it /mnt/containerstore) in the host to container to /nix/store, so that mount events are propagated. In this directory every packages, which is used by container, can be bind mounted on the host to /mnt/containerstore. This approach has the advantage that containers can be updated without performing a restart. I plan to build something like this for lxc actually.

[grahamc]

FWIW OverlayFS is not POSIX-complete: (search for POSIX here:) https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/

[jpotier]

One interesting use case would be the deployment of web-based applications such as wordpress, dokuwiki, owncloud, nextcloud, etc. We can package a fixed version of their source and mount them with overlayFS for each instance we want to spawn. That should make upgrade easy, as well as giving a sane default to every containers while not including php-fpm, lighttpd, database drivers, etc. which are still managed by nix.

[mmahut]

Any news on this issue?

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[spacekitteh]

Still important

[stale[bot]]

I marked this as stale due to inactivity. → More info