search

pdtpartners / nix-snapshotter

Brings native understanding of Nix packages to containerd
MIT License
532 stars 15 forks source link

k3s example: error pulling image #102

Closed antoinerg closed 7 months ago

antoinerg commented 11 months ago

First of all, thanks for the great project! I've always wanted K8S to be declarative all the way down to container images. This is awesome!

I tried running the examples on k3s by commenting out ./kubernetes.nix in favor of ./k3s.nix in the following file: https://github.com/pdtpartners/nix-snapshotter/blob/6eb21bd3429535646da4aa396bb0c1f81a9b72c6/modules/nixos/vm.nix#L11-L12

then ran nix run ".#vm" to get a VM.

Although it successfully boots a VM with k3s, I hit an error when pulling a Nix image like the preloaded one (eg. kubectl apply -Rf /etc/kubernetes/redis will result in a pod that fails with a pull image error).

Doing the same steps as above but using ./kubernetes.nix works flawlessly.

antoinerg commented 8 months ago

I think the reason why this was failing is because we need to use a k3s version that includes the changes that closed issue https://github.com/k3s-io/k3s/issues/8480

antoinerg commented 8 months ago

I think the reason why this was failing is because we need to use a k3s version that includes the changes that closed issue k3s-io/k3s#8480

... and we then need to point k3s's kubelet to use nix-snapshotter image service:

services.k3s = {
  enable = true;
  configPath = pkgs.writeText "k3s_config.yaml" ''
    container-runtime-endpoint: "unix:///run/containerd/containerd.sock"
    image-service-endpoint: "unix:///run/nix-snapshotter/nix-snapshotter.sock"
  '';
};
antoinerg commented 8 months ago

Along with https://github.com/pdtpartners/nix-snapshotter/issues/102#issuecomment-1880169796, the fix from https://github.com/pdtpartners/nix-snapshotter/pull/107 is required!

antoinerg commented 8 months ago

Along with #102 (comment), the fix from #107 is required!

The fix landed so I can now follow-up with a PR with the changes from https://github.com/pdtpartners/nix-snapshotter/issues/102#issuecomment-1880169796 :tada:

antoinerg commented 7 months ago

The image-service-endpoint flag was added to k3s in patch release 1.27.7 (reference: https://github.com/k3s-io/k3s/issues/8481). Unfortunately, nixpkgs/nixos-23.11 uses v1.27.6.

RobbieBuxton commented 7 months ago

The image-service-endpoint flag was added to k3s in patch release 1.27.7 (reference: k3s-io/k3s#8481). Unfortunately, nixpkgs/nixos-23.11 uses v1.27.6.

We could patch it with an overlay until we decide to move onto the next stable Nixos release in the future?