Better UX for using local registry

[h7kanna]

What would you like to be added:

Add a special node to the Kind cluster with 'registry' role. This node can run docker registry without the user having to run any extra scripts to configure Kind cluster

Why is this needed:

This can be an alternative to 'kind load docker-image' and naturally integrates with 'docker push' CI workflows.

Also, the registry storage can be mounted from the host and kind load can be avoided for easier setup.

[BenTheElder]

We're still designing this, there are some tradeoffs that depend on how the networking changes to fix https://github.com/kubernetes-sigs/kind/issues/148

The obvious "registry" role route has some downsides, including that you'll need to push to each cluster.

[diafour]

I choose not to run a local registry in-cluster. It is not great for testing against multiple versions and/or clusters. So I run local registry as a standalone docker container with mounted volume in a home directory. Kind cluster is started with a small script, that does two things after the cluster is ready: it adds localhost:5000 as a trusted registry to /etc/containerd/config.toml and creates a NodePort Service and a DaemonSet to proxying from that NodePort to docker host IP.

That is an adopted solution back from kubeadm-dind-cluster days: https://github.com/kubernetes-retired/kubeadm-dind-cluster/issues/56#issuecomment-387463386

[BenTheElder]

thanks for the input!

... it adds localhost:5000 as a trusted registry to /etc/containerd/config.toml ...

you can do this at boot time now, xref: https://kind.sigs.k8s.io/docs/user/local-registry/

... and creates a NodePort Service and a DaemonSet to proxying from that NodePort to docker host IP. ...

That's a neat approach, thanks :-)

[diafour]

Yeah, that config.toml override was for 0.5 release. Thanks for the link!

[diafour]

Hooray! I've just realized that endpoint should not be the same as mirror name and simplified my setup: no proxying with DaemonSet required, just containerdConfigPatches: https://gist.github.com/diafour/13cef191b7cf39543393d310dd6353a0

The workflow is like this:

docker build -t "localhost:5000/name:tag" .
docker push localhost:5000/name:tag

kind: Pod
...
    image: localhost:5000/name:tag

cluster-15.sh create
kubectl replace ... 

There are create and delete commands to address #148 ;)

@BenTheElder Thanks again!

[nicks]

Hi! I work on Tilt! We care so much about good local registry support that if you're not using a local registry, we emit a warning that you should be.

We currently point people at this script to set it up: https://github.com/windmilleng/kind-local/blob/master/kind-with-registry.sh but we'd love to recommend a solution built into Kind.

I think the main things that are important to us (from a UX perspective) are (in order): 1) The ability for tooling to auto-detect the registry from the cluster 2) The ability to delete the cluster and create a new one without deleting the registry

So I think our ideal CLI UX would be something like:

kind create registry [--name optional-name]
kind create cluster --with-registry default

where --with-registry default annotated the Kubernetes nodes with the URL of the registry (as the kind-with-registry.sh script above does)

[BenTheElder]

still on my radar! we've mostly landed the host reboot work which includes using a docker network w/ embeded DNS, but had to spend time on a number of other critical fixes.

we still need some follow up to that, but once we settle down the new networking a bit we should have a better idea what we can build the registry integration against.

[BenTheElder]

I think instead of a "create registry" w/ ordering issues (and presumably failing if it already exists etc.) the ux from k3d is closer to what people need, ie. --enable-registry when creating the cluster. (which enables a shared local registry). https://github.com/rancher/k3d/blob/master/docs/registries.md#using-the-k3d-registry

[BenTheElder]

(and I've just filed what I hope is the last major fix related to the DNS stuff, we neglected to update the NO_PROXY logic to handle it properly)

[Morgma]

Hooray! I've just realized that endpoint should not be the same as mirror name and simplified my setup: no proxying with DaemonSet required, just containerdConfigPatches: https://gist.github.com/diafour/13cef191b7cf39543393d310dd6353a0

The workflow is like this:

docker build -t "localhost:5000/name:tag" .
docker push localhost:5000/name:tag

kind: Pod
...
    image: localhost:5000/name:tag

cluster-15.sh create
kubectl replace ... 

There are create and delete commands to address #148 ;)

@BenTheElder Thanks again!

@diafour, I'm using Docker for Windows running on WSL 2, Ubuntu 18.04. When I run kind-with-registry.sh, I can push/pull from my host at localhost:5000, however, k8s fails to pull images from the registry from that endpoint. I've tried substituting "host.docker.internal", but container pulls still fail to resolve the endpoint. Am I missing some specific guidance for Docker on WSL2?

Thanks!

EDIT:

Somewhere along the line either I wasn't running 'docker network connect...' or I wasn't annotating the node with "kind.x-k8s.io/registry=localhost:5000". My image pulls are resolving now.

[MadhavJivrajani]

/remove-kind design /kind feature kind/design is migrated to kind/feature, see https://github.com/kubernetes/community/issues/6144 for more details

[asilverman]

Consider using a different port than 5000 since its used by Apple MacOS Monterrey

[BenTheElder]

That is wildly unfortunate, 5000 is a commonly used development port and docker special cases allowing localhost:5000 as an HTTP (not HTTPS) endpoint out of the box.

Reconfiguring dockerd is not something kind ought to be doing and will introduce user friction.

Not excited to make the experience worse on other platforms because Apple decided to suddenly start binding to a commonly used port OOTB :/

[diafour]

Just for the record, there is no special case localhost:5000. dockerd trusts all local IPs despite the port:

Local registries, whose IP address falls in the 127.0.0.0/8 range, are automatically marked as insecure as of Docker 1.3.2. It is not recommended to rely on this, as it may change in the future.

Insecure registries

[BenTheElder]

Well I stand corrected 🙃 , thank you for sharing @diafour 🙏

If someone wants to update the port in the sample script, that should be a trivial change then 🙃

[HummingMind]

Hello! Is this still being worked on? If so, any ETA? I see it was put back into the backlog. :(

Thank you!

[BenTheElder]

I’m sorry but we’re prioritizing bug fixes like fixing reboot support and keeping up with Kubernetes, and user support, it doesn’t leave much time for designing / reviewing / implementing new features at the moment.

[HummingMind]

I’m sorry but we’re prioritizing bug fixes like fixing reboot support and keeping up with Kubernetes, and user support, it doesn’t leave much time for designing / reviewing / implementing new features at the moment.

That is fair. Hopefully that changes in the near future. Thank you for the work you guys do on KIND!

[BenTheElder]

Thanks, the core maintainers work primarily on Kubernetes but we're trying 😅

Smaller updates:

• The breaking change in https://github.com/kubernetes-sigs/kind/issues/2875 is problematic for even the simple script, we're going to need to figure out a migration path.
• We did move the sample script to port 5001 after above discussion about macOS and insecure exception https://github.com/kubernetes-sigs/kind/pull/2621

[colin-williams]

With or without using a local registry, would it be possible to create a configuration setting to "proxy" container images automagically through the local docker host?

That way, bringing up or taking down kind clusters we'd still have the images locally. This would mitigate useless downloads that waste bandwidth, time, and energy. As a bonus it would be nice if the image didn't have to be duplicated between kind and the local docker store.

[BenTheElder]

With or without using a local registry, would it be possible to create a configuration setting to "proxy" container images automagically through the local docker host?

Long story short: not readily, but it should be technically feasible if someone wished to invest in it.

Currently the KIND maintainers are pretty focused on bug-fixes and maintenance when we're working on KIND (and not say, Kubernetes core).

Containerd does have a way to configure * host to a mirror, except they changed the mirroring config (https://github.com/kubernetes-sigs/kind/issues/2875, https://github.com/kubernetes-sigs/kind/pull/2952) in a way that may not permit that anymore?, and you'd need a registry that implemented a recent-ish OCI distribution header (https://github.com/containerd/containerd/issues/3734) to detect the real target host in the pull and mirror all hosts. I'm not aware of such an implementation yet, previously this wasn't feasible because the server didn't know what original host the target image was being mirror-pulled from.

As a bonus it would be nice if the image didn't have to be duplicated between kind and the local docker store.

Docker does not document or expose the local docker image storage except through pull/push/save/load, all of which do not match registries in that they are single-architecture (IE no manifest lists, which docker can do, but only through a special subcommand with different storage, awkwardly). We also support podman, and nominally support remote dockerd.

[nicks]

FWIW, there's work actively under way to switch Docker's image store to use containerd, though it's still in beta - https://www.docker.com/blog/extending-docker-integration-with-containerd/

Docker has talked a little bit about how we might make it possible to auto-magically make the images appear in Kind. As Ben said, it's theoretically possible but there are a lot of devils hiding in the details. Might have to try a few different approaches.

[BenTheElder]

FWIW, there's work actively under way to switch Docker's image store to use containerd, though it's still in beta - https://www.docker.com/blog/extending-docker-integration-with-containerd/

That's neat but it still won't be the node level containerd, most of that metadata is in a boltdb that you can't share between instances. (Also we can't just read files from the kind binary POV since they may be on different machines) not to mention podman which has its own storage.

The only approach I see here as semi-feasible is if docker and podman etc (see also requests for nerdctl) get a special image pull proxy that reads the above header and then checks local storage but that's kinda fraught with nasty details (like how do you force pulling a new version of a tag)

If you're willing to pay the cost and just want to cache between runs without knowing the images and explicitly storing / loading / mirroring them, you can use an extraMount and mount the containerd storage through to a directory on the host. However this will grow unbounded as for other reasons we have no image GC enabled in kind (which has it's own pile of issues that I won't dive into here)

[Nossnevs]

One thing that should at least be added to the page https://kind.sigs.k8s.io/docs/user/local-registry/ is how to handle proxy setting.

If you have set the http_proxy environment variable, you have to add kind-registry to no_proxy variable. If you don't then it will try to use the proxy to find it. Not the easiest to find because I for example got ErrImagePull forbidden in Kubernetes.

[BenTheElder]

We're using that registry container image in our example docs. The issue is about the user experience around making it accessible to the nodes and the host.