search

pdtpartners / nix-snapshotter

Brings native understanding of Nix packages to containerd
MIT License
532 stars 15 forks source link

Add plugin package for embedding nix-snapshotter in containerd #112

Closed elpdt852 closed 7 months ago

elpdt852 commented 7 months ago

Due to rootless k3s being in its own namespaces (via rootlesskit), it's not possible for an external rootless containerd to be set up with rootless k3s. The only way forward is adding support for nix-snapshotter to k3s upstream by providing a builtin plugin that can be compiled with the containerd embedded in k3s.

This PR provides the plugin package that allows compiling containerd with nix-snapshotter support.

It also includes a commit that has a k3s patch to run with nix-snapshotter support. Since k3s derivation is very difficult to overlay, we include the derivations in-tree to make the changes locally. Once we land support in upstream k3s this can all be deleted.

k3s changes for nix-snapshotter support: https://github.com/hinshun/k3s/commit/9b1247f6b5d0374abe846f92a60b1653e18eefa9

elpdt852 commented 7 months ago

We can depend on ${nixpkgs}/.../path/to/builder.nix but in general it's bad practice because whenever nixpkgs bumps (even with unrelated changes), the input hash becomes different and k3s needs to be rebuilt from scratch again (it's pretty slow). Ideally we should do it in an overlay, but I tried and it was exceedingly difficult especially when you modify go.mod in the patch because you also need to rebuild the go-modules vendor directory.

In my opinion, since this is short-lived anyway, I'd like to keep it dead simple and just copy & paste. We should file an issue to nixpkgs about making overriding k3s possible.

elpdt852 commented 7 months ago

The changes to k3s I've maintain in a fork here: https://github.com/hinshun/k3s/commit/9b1247f6b5d0374abe846f92a60b1653e18eefa9

We should avoid reviewing it in the fork or the patch in this PR, instead review it in the upstream k3s PR: https://github.com/k3s-io/k3s/pull/9319