Document using Spegel with pull through registries

[phillebaba]

After carefully reviewing the following containerd-related errors, I deleted the configurations related to "mirror" in /etc/rancher/k3s/registries.yaml and /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl. Now, spegel is working properly.

time="2023-11-20T23:28:55.907708354+08:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="invalid plugin config: `mirrors` cannot be set when `config_path` is provided"

That's a bit unfortunate since the docs state:

Spegel does not aim to replace projects like Harbor or Zot but instead complements them.

I was hoping to use spegel in my cluster and zot as a pull thru cache deployed elsewhere. For what it's worth this is my current containerd mirrors:

mirrors:
  docker.io:
    endpoint:
      - https://zot.domain.tld/v2/docker.io
  ghcr.io:
    endpoint:
      - https://zot.domain.tld/v2/ghcr.io
  quay.io:
    endpoint:
      - https://zot.domain.tld/v2/quay.io
  gcr.io:
    endpoint:
      - https://zot.domain.tld/v2/gcr.io
  registry.k8s.io:
    endpoint:
      - https://zot.domain.tld/v2/registry.k8s.io
  public.ecr.aws:
    endpoint:
      - https://zot.domain.tld/v2/public.ecr.aws

I don't see a way to have spegel take over this responsibility it seems like you either have spegel or a pull thru cache, maybe this can be a feature request?

Originally posted by @onedr0p in https://github.com/XenitAB/spegel/issues/212#issuecomment-1830879118

[phillebaba]

@onedr0p I created a new issue for this as it is not realted to k3s, and can be fixed with more documentation.

[onedr0p]

Thanks @phillebaba, in the meantime do you have any tips on how to get that working? I don't mind contributing to the docs if I can get it working on my end.

[phillebaba]

Could you check the docs added to #280 and see if this is enough?

[onedr0p]

@phillebaba I thought about doing it that way to begin with but how does spegel know how to fetch the container from my zot paths as defined in my containerd config above which tells containerd to use (e.g.) https://zot.domain.tld/v2/docker.io if the mirror is for docker.io

The config I have for zot looks like this which was crafted by using their docs on setting up a mirroring.

[onedr0p]

To explain a bit further, I am using a zot as a transparent proxy with that containerd config, so to expand a bit I am unsure about the following being set in the additionalMirrorRegistries

spegel:
  additionalMirrorRegistries:
    # not sure if this works since zot is configured via paths to handle the individual registries
    - https://zot.example.com
    # how would spegel know to pull on dockerhub images from here?
    - https://zot.example.com/v2/docker.io
    # how would spegel know to pull on ghcr images from here?
    - https://zot.example.com/v2/ghcr.io

[phillebaba]

Zot seems to manage pull through caches in a similar way to Harbor. I have never been a fan of this solution as it changes the original registry endpoint. The mirror configuration component of Spegel is not a critical component of this project and is more of a helper to simplify setup of Spegel. You could in theory disable this and create your own mirror configuration.

This is probably a challenge for most people using a pull through cache no matte the flavor, so it needs to be fixed. I will have to think about the best configuration option here as I do not want to redo it in another 6 months.

My idea is to add a boolean called rewritePath or something similar which would setup a mirror configuration the way Zot wants it for these additional registries.

[ChristianCiach]

@phillebaba I don't know if this is related, but I just want to point out how K3s' fork of Containerd handles these cases. They especially added support for regristry rewrites that we are using for exactly this use case, so that we can use our Harbor instance as a pull-through cache for docker hub. See the documentation here: https://docs.k3s.io/installation/private-registry#rewrites

Unfortunately they never managed to get this merged upstream at Containerd.

[onedr0p]

@ChristianCiach it looks like spegel might be baked into k3s as a optional feature which probably covers that use-case (for k3s anyways).

See: https://github.com/k3s-io/k3s/pull/8977

[1337andre]

Hey folks, any update on this?