Failed to pull image, content digest sha256 not found

[pizzapim]

Hello! First I would like to thank you for this awesome project! I have come across an issue while using it.

Context: I am running a k3s cluster on top of NixOS. I have already deployed nix-snapshotter successfully on my nodes. I can confirm my setup works as I was able to pull and run a Redis image from the Nix store.

However, I am now encountering a problem with a particular Docker image, mpepping/cyberchef. More specifically, I am using the image created by:

pkgs.dockerTools.pullImage {
  finalImageName = "mpepping/cyberchef";
  finalImageTag = "latest";
  imageDigest = "sha256:5044c72dd8070fb6e0595e720fc4440bf6168493b2433db06a1c966406398ba2";
  imageName = "mpepping/cyberchef";
  sha256 = "177yjfbz0ijc8lfqfr50fhqqmjk72373c0igyrxv3wwg0pyrgpv4";
}

When I use the Nix store path to this image, I receive the following events on the Kubernetes pod:

  Normal   Scheduled  19s   default-scheduler  Successfully assigned default/cyberchef-65769f8c78-7cmxv to lewis
  Normal   Pulling    19s   kubelet            Pulling image "nix:0/nix/store/frpcl21jwz0zhr2whmi21affqbfn9dqw-docker-image-mpepping-cyberchef-latest.tar"
  Warning  Failed     12s   kubelet            Failed to pull image "nix:0/nix/store/frpcl21jwz0zhr2whmi21affqbfn9dqw-docker-image-mpepping-cyberchef-latest.tar": content digest sha256:1ca8358066425ba22dd40a3bb562a1be846e7a721255ddfedcfb8468b028308f: not found
  Warning  Failed     12s   kubelet            Error: ErrImagePull
  Normal   BackOff    11s   kubelet            Back-off pulling image "nix:0/nix/store/frpcl21jwz0zhr2whmi21affqbfn9dqw-docker-image-mpepping-cyberchef-latest.tar"

The logs of nix-snapshotter show no problem:

msg="[image-service] Loading nix image archive"
msg="Importing image" ref="docker.io/mpepping/cyberchef:latest"
msg="Creating image" ref="docker.io/mpepping/cyberchef:latest"
msg="Created image" ref="docker.io/mpepping/cyberchef:latest"

It does look like this is an upstream containerd issue: https://github.com/containerd/containerd/issues/9873 Do you have any ideas how to avoid this issue?

[elpdt852]

So pkgs.dockerTools.pullImage produces a regular image, and nix-snapshotter is able to run those, but doesn't know how to pull them. I don't think it's possible to use nix-snapshotter as an image service (i.e. using nix:0/nix/store/...) to resolve those, so you will need to specify the reference in your kubernetes deployment to point to docker.io/mpepping/cyberchef:latest directly, or build the image using pkgs.nix-snapshotter.buildImage.

[elpdt852]

That said, nix-snapshotter image service could be potentially extended to pull regular images via the nix:0 path, but not currently supported.

[pizzapim]

Thanks for your response! I see, the images pulled using pkgs.dockerTools.pullImage are indeed not OCI compliant. I've found a partial workaround by passing the pulled Docker image to nix-snapshotter.buildImage in the fromImage attribute. E.g.:

  pkgs.nix-snapshotter.buildImage {
    name = "cyberchef";
    resolvedByNix = true;
    fromImage = dockerImage;
  };

While that copies the files of the image, it does not copy the configuration of it. There is probably a smarter way, but I created a small function to get the configuration of a Docker image:

  getDockerImageConfig = dockerImage:
    let
      configJson = pkgs.runCommand "config.json"
        {
          nativeBuildInputs = [ pkgs.skopeo pkgs.jq ];
        }
        ''
          skopeo --tmpdir $TMPDIR --insecure-policy inspect docker-archive:${dockerImage} --config | jq '.config' > $out
        '';
    in
    builtins.fromJSON (builtins.readFile configJson);

And I just pass it like this to the bulidImage function:

  pkgs.nix-snapshotter.buildImage {
    name = "cyberchef";
    resolvedByNix = true;
    fromImage = dockerImage;
    config = getDockerImageConfig dockerImage;
  };

This seems to work at least for the docker.io/mpepping/cyberchef:latest image!

[bglgwyng]

That said, nix-snapshotter image service could be potentially extended to pull regular images via the nix:0 path, but not currently supported.

Would you happen to have any idea how to do it? Could you please share the outline? I think pulling it from that machine is possible when the derivation is already built in some other machine. However, when it's not found in any other machine, how can we trigger a new build from the current nix:0 path?

[elpdt852]

Hi @bglgwyng I answered more in the new issue you opened here: https://github.com/pdtpartners/nix-snapshotter/issues/147#issuecomment-2344591689

To be clear, nix-snapshotter does not build derivations on-demand and works with pre-built Nix store paths. These could be substituted on-demand but they operate at the Nix store path level, not Nix derivations (or Nix expressions).

That said, you can evaluate, build and deploy a nix:0 image to Kubernetes as demonstrated in the demo VMs because its fully declarative, but the "build on-demand" (and evaluation) is a feature of Nix, and nix-snapshotter itself doesn't understand derivations or Nix expressions.