Getting gardener-extension-registry-cache production ready

[oliver-goetz]

How to categorize this issue?

/area quality cost /kind task

What would you like to be added: At the end of our Hackathon we had some tasks on your list in order to make the registry-cache production ready.

Why is this needed: Getting the registry-cache extension production ready.

Steps

• [x] Refactor current approach to change containerd config files
  • [x] Use the new way of configuring mirrors for containerd. The new way is https://github.com/containerd/containerd/blob/main/docs/hosts.md. The old (current) way is https://github.com/containerd/containerd/blob/main/docs/cri/registry.md#configure-registry-credentials: https://github.com/gardener/gardener-extension-registry-cache/pull/27
  • [x] Replace the DaemonSet approach by OperatingSystemConfig mutating webhook. The webhook adds new containerd configuration files under /etc/containerd/cert.d/: https://github.com/gardener/gardener-extension-registry-cache/pull/27
  • [x] Introduce a dependency in the Shoot reconciliation flow such as - the OperatingSystemConfig deploy should be after the Extension resources are ready. In this way an extension such as registry-cache can first deploy the registries to the Shoot, then mutate the OperatingSystemConfig with information gathered after the Extension is ready (the Service IP of the registry Services). https://github.com/gardener/gardener/pull/8232
  • [x] Adapt the containerd-initializer unit in gardener/gardener to configure set the config_path: https://github.com/gardener/gardener/pull/8094 Modify the default containerd config in gardener to specify

    [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"

  • [x] Make sure that there are no side effects of configuring empty hosts dir for containerd (ref https://github.com/containerd/containerd/blob/main/docs/hosts.md)
  • [x] Graduate the ContainerdRegistryHostsDir feature gate to GA. Clean up the feature gate
    • https://github.com/gardener/gardener/pull/8873
    • https://github.com/gardener/gardener/pull/8979
    • https://github.com/gardener/gardener/pull/9058
  • [x] Clean up migration code for provider-local in gardener/gardener:
    • https://github.com/gardener/gardener/pull/8441
    • https://github.com/gardener/gardener/pull/8442
  • [x] Implement orphan file deletion in the cloud-config executor. When the registry-cache extension stops mutating the OperatingSystemConfig, the registry config files will leak on the host. With such orphan file deletion mechanism, cloud-config executor will take care to delete files which are no longer present in the OperatingSystemConfig. According to the docs, updates under the hosts config dir (for example /etc/containerd/cert.d/) do not require restart of containerd.
  • [x] gardener/gardener: https://github.com/gardener/gardener/pull/8227
  • [x] Vendor gardener/gardener@v1.76.0 in gardener-extension-os-gardenlinux - https://github.com/gardener/gardener-extension-os-gardenlinux/pull/118, released with v0.21.0
  • [x] Vendor gardener/gardener@v1.76.0 in gardener-extension-os-suse-chost - https://github.com/gardener/gardener-extension-os-suse-chost/pull/101, released with v1.23.0
  • [x] Vendor gardener/gardener@v1.76.0 in gardener-extension-os-ubuntu - https://github.com/gardener/gardener-extension-os-ubuntu/pull/88, released with v1.23.0
  • [x] Vendor gardener/gardener@v1.76.0 in gardener-extension-os-coreos and do the required adaptation - https://github.com/gardener/gardener-extension-os-coreos/pull/70, released with v1.18.0
  • Using the /etc/containerd/conf.d/ imported files is flawed and doesn't work for all cases: https://github.com/gardener/gardener/blob/master/docs/usage/custom-containerd-config.md / https://github.com/gardener/gardener/pull/7316 / https://github.com/containerd/containerd/issues/5837. That's why we cannot use the custom containerd config import functionality for now.
  • [x] Node bootstrap problem: https://github.com/gardener/gardener-extension-registry-cache/pull/68
• [x] Node machines cannot resolve the DNS names of the registry cache services yet. Currently, containerd config includes IP addresses of the registry cache pods/services. Find a way to make this DNS resolution work.
• [x] Support container registries which require credentials: https://github.com/gardener/gardener-extension-registry-cache/pull/94
  • [x] Add e2e/testmachinery tests: https://github.com/gardener/gardener-extension-registry-cache/pull/184
  • [x] Document the limitation why we have single credentials per pull through cache instance: https://github.com/gardener/gardener-extension-registry-cache/pull/152
• [x] Tests
  • [x] Add unit tests: https://github.com/gardener/gardener-extension-registry-cache/pull/36
  • [x] Add testmachinery tests: https://github.com/gardener/gardener-extension-registry-cache/pull/38
  • [x] Add e2e tests
  • https://github.com/gardener/gardener-extension-registry-cache/pull/30
  • https://github.com/gardener/ci-infra/pull/847
  • [x] Fix make extension-down and rework skaffold handling to also deploy the registry-cache admission during make extension-up: https://github.com/gardener/gardener-extension-registry-cache/pull/40
• [x] Add documentation
  • [x] Add documentation for end users: https://github.com/gardener/gardener-extension-registry-cache/pull/44
  • [x] Add documentation for developers (architectural design, internal workings): https://github.com/gardener/gardener-extension-registry-cache/pull/81
  • [x] Add validation for immutability of the disk size. Currently the corresponding field in the StatefulSet is immutable. There is a dedicate task for the resize but we need to agree how we want to support it: https://github.com/gardener/gardener-extension-registry-cache/pull/42
  • [x] Add validation to prevent adding multiple registry-cache extensions with the same upstream: https://github.com/gardener/gardener/issues/8446
  • [x] Forbid caches with the same upstream: https://github.com/gardener/gardener-extension-registry-cache/pull/45
• [x] Double-check what happens if disk size of registry is insufficient (consider activating garbage collection or similar solution)
  • The garbage collection working is described in the docs PR (https://github.com/gardener/gardener-extension-registry-cache/pull/44)
• [x] Redesign the garbage collection API: https://github.com/gardener/gardener-extension-registry-cache/pull/144
  • Expose ttl and remove the existing option to enable/disable the garbage collection. Consider forbidding ttl < 24h as ttl=0 can mean that image never gets garbage collected (double check this).
  • Drop the existing garbageCollection.enabled field in favor of the garbageCollection.ttl field.
  • [x] Change the garbageCollectionEnabled field to garbageCollection.enabled. This is to make possible adding new fields related to the garbage collection in future (the ttl field): https://github.com/gardener/gardener-extension-registry-cache/pull/53
• [x] Add monitoring (https://docs.docker.com/registry/configuration/)
  • [x] (upstream contribution) Make available the registry-cache metrics in the /metrics endpoint that is prometheus native format
    • https://github.com/distribution/distribution/pull/4045
    • https://github.com/distribution/distribution/pull/4047, https://github.com/distribution/distribution/pull/4156
    • [x] (upstream contribution) Init upstream metrics with 0 vector: https://github.com/distribution/distribution/pull/4283
  • [x] Decide on which namespace the extension will be deployed - registry-cache or kube-system. The control plane monitoring currently does not monitor workload from the registry-cache namespace (i.e. we don't have metrics for Pod cpu/memory usage; logs are not collected): https://github.com/gardener/gardener-extension-registry-cache/pull/52
  • [x] Add dashboard with:
    • [x] registry cache panels: https://github.com/gardener/gardener-extension-registry-cache/pull/110 initialize counters: https://github.com/distribution/distribution/pull/4283
    • [x] volume panels: https://github.com/gardener/gardener-extension-registry-cache/pull/112
  • [x] Check for kubelet metrics to show a panel about image pull times: https://github.com/gardener/gardener/pull/9422
  • [x] Aggregate the registry-cache metrics: https://github.com/gardener/gardener-extension-registry-cache/pull/169
• [x] Add alerting for the registry-cache volume size: https://github.com/gardener/gardener-extension-registry-cache/pull/96
  • [x] Scrape the required kubelet metrics in gardener/gardener: https://github.com/gardener/gardener/pull/8798
• [x] Allow configuring containerd mirrors without deploying the caches to the shoot (add option to providerConfig and corresponding handling in controller): https://github.com/gardener/gardener-extension-registry-cache/pull/134
  • [x] Update docs in gardener/gardener: https://github.com/gardener/gardener/pull/9165
• [x] Switch the registry-cache extension to do not use deprecated containerd configs
  • [x] [prerequisite] Adapt the kind clusters in gardener/gardener to do not use the deprecated keys: https://github.com/gardener/gardener/pull/8047
  • [x] [prerequisite] Adapt the Shoot cluster in provider-local to do not use the deprecated keys: https://github.com/gardener/gardener/pull/8094
• [x] Check and if possible reduce the resources requests and limits for the prow job definition that builds the registry-cache container image (gardener-extension-registry-cache-test-builds and gardener-extension-registry-cache-build-images): https://github.com/gardener/ci-infra/pull/795
• [x] cri-config-ensurer and registry images are wrong: https://github.com/gardener/gardener-extension-registry-cache/pull/23
• [x] Deletion of a Shoot cluster with enabled registry-cache doesn't work. gardenlet in the Cleaning Kubernetes resources deletes the registry-cache resources which are recreated by GRM in few moments: https://github.com/gardener/gardener-extension-registry-cache/pull/25
• [x] PVC and StorageClass creation is racy. It can be the case that the PVC is created first before the StorageClass. Then the PVC hangs in Pending state with reason no persistent volumes available for this claim and no storage class is set: https://github.com/gardener/gardener-extension-registry-cache/pull/51
• [x] Fix the --version command: https://github.com/gardener/gardener-extension-registry-cache/pull/98
• [x] Switch to the REUSE license format: https://github.com/gardener/gardener-extension-registry-cache/pull/146
• [x] Support make extension-up in provider-extensions setup @dimitar-kostadinov https://github.com/gardener/gardener-extension-registry-cache/pull/193
• [x] skaffold does not build and update on changes to the images.yaml file: https://github.com/gardener/gardener-extension-registry-cache/pull/57
• [x] Allow reuse of the logic in gardener/gardener that checks for skaffold ko dependencies: https://github.com/gardener/gardener/pull/8630
• [x] Go through the component-checklist in gardener/gardener and double-check every item: https://github.com/gardener/gardener-extension-registry-cache/pull/85
  • [x] Add readiness and liveness probe to the extension: https://github.com/gardener/gardener-extension-registry-cache/pull/31
  • [x] Add readiness and liveness probe to the registry Pods: https://github.com/gardener/gardener-extension-registry-cache/pull/55
  • [x] Drop the healthcheck controller as after https://github.com/gardener/gardener/pull/7462 gardenlet checks for all ManagedResources in the Shoot control plane. We don't need anymore the healthcheck controller: https://github.com/gardener/gardener-extension-registry-cache/pull/32
  • [x] Add PriorityClass for the registry-cache Pod: https://github.com/gardener/gardener-extension-registry-cache/pull/47
  • [x] Provide resource requirements: https://github.com/gardener/gardener-extension-registry-cache/pull/71
  • [x] Define a VerticalPodAutoscaler: https://github.com/gardener/gardener-extension-registry-cache/pull/71
  • [x] Define RBAC roles with minimal privileges: https://github.com/gardener/gardener-extension-registry-cache/pull/85
  • [x] Use NetworkPolicys to restrict network traffic: https://github.com/gardener/gardener-extension-registry-cache/pull/85
  • [x] Do not run components in privileged mode: https://github.com/gardener/gardener-extension-registry-cache/pull/85
  • [x] Choose the proper Seccomp profile: https://github.com/gardener/gardener-extension-registry-cache/pull/85
• [x] Review all registry config options before we move to production (ref https://distribution.github.io/distribution/about/configuration/):
  • [x] https://github.com/distribution/distribution/issues/4304: https://github.com/distribution/distribution/pull/4305
• [x] Check https://github.com/containerd/containerd/issues/4861 and see whether with docker there are timeouts of 30 seconds
• [x] Add support for /test pull-gardener-publish-test-images command for the registry-cache repo so that it is possible to publish images from PRs: https://github.com/gardener/ci-infra/pull/970
• [x] Beautify the imageVectorOverwrite so that it is possible to pass it as YAML, not as a single line string: https://github.com/gardener/gardener-extension-registry-cache/pull/80
• [x] Make the storageClassName configurable: https://github.com/gardener/gardener-extension-registry-cache/pull/101
• [x] Remove the deprecated v1alpha1 API version: https://github.com/gardener/gardener-extension-registry-cache/pull/141
  • [x] Adapt the e2e and testmachinery tests to use v1alpha2 API version only: https://github.com/gardener/gardener-extension-registry-cache/pull/140
• [x] Fix CPM for a Shoot with registry-cache extension enabled: https://github.com/gardener/gardener-extension-registry-cache/pull/114
• [x] Fix/mitigate https://github.com/distribution/distribution/issues/4191: https://github.com/gardener/gardener-extension-registry-cache/pull/128
• [x] Adjust the StatefulSet naming convention shorter to accommodate longer registry name: https://github.com/gardener/gardener-extension-registry-cache/issues/120: https://github.com/gardener/gardener-extension-registry-cache/pull/129
• [x] Think about mitigation for https://github.com/distribution/distribution/issues/4249: https://github.com/gardener/gardener-extension-registry-cache/pull/131
• [x] Check if we can speed up the configure-containerd-registries.service unit by using another retry strategy in the unit other than exponential backoff: https://github.com/gardener/gardener-extension-registry-cache/pull/137
• [x] Upgrade distribution/distribution to v3.0.0-alpha.1: https://github.com/gardener/gardener-extension-registry-cache/pull/138
  • [x] Copy the DockerHub image to AR: https://github.com/gardener/ci-infra/pull/1128
  • https://github.com/distribution/distribution/issues/4270
• [x] Drop vendoring from the registry-cache repo: https://github.com/gardener/gardener-extension-registry-cache/pull/147
• [x] Fix e2e tests flakes: https://github.com/gardener/gardener-extension-registry-cache/pull/196
• [x] Investigate how to reproduce and fix https://github.com/distribution/distribution/issues/2374: https://github.com/distribution/distribution/pull/4293, https://github.com/gardener/gardener-extension-registry-cache/pull/196
  • [x] Activate registry graceful shutdown: https://github.com/gardener/gardener-extension-registry-cache/pull/162
• [x] Remove the deprecated v1alpha2 API version: https://github.com/gardener/gardener-extension-registry-cache/pull/165
  • [x] Adapt the e2e and testmachinery tests to use v1alpha3 API version only: https://github.com/gardener/gardener-extension-registry-cache/pull/151
• [x] Support upstreams with http:// and port (optional): https://github.com/gardener/gardener-extension-registry-cache/pull/183
• [x] Long upstreams will be rejected due to the label restriction of 63 chars: https://github.com/gardener/gardener-extension-registry-cache/pull/186
• [x] (upstream contribution) Open issue to the upstream to change the base image of the registry image to distroless: https://github.com/distribution/distribution-library-image/issues/168
  • [x] (prerequisite) Rework the e2e tests to use the registry API instead of the scheduler-state.json file https://github.com/gardener/gardener-extension-registry-cache/pull/196
• [x] Prepare a blog: https://github.com/gardener/documentation/pull/495

---

• [x] Evaluate dragonfly: https://github.com/dimitar-kostadinov/gardener-extension-registry-cache/tree/dragonfly-eval

[JensAc]

Seems to be a reasonable list. However, I am wondering whether we could just release a v0.0.1 before. Then we (23T) could easily include the extension in our public gardener installation and just see how the current state and further developments behave in a running environment. Any comments on that?

[JensAc]

Would you like to release the v0.0.1 @oliver-goetz? Actually, I am not entirely sure how what kind of release workflow is intended for this repo.

[oliver-goetz]

@JensAc sorry I missed your question 😅 There is no release workflow yet. I'll think about it. However, we are already building dev images for each commit on main. You can find the images names + tags in the jobs log.

[rfranzke]

https://github.com/gardener/gardener/pull/6999 is merged now which (I think) was the prerequisite to follow the "webhook approach" :)

[oliver-goetz]

@JensAc finally, there is the first release 😄

[JensAc]

Nice! Many thanks :+1:

[timebertt]

Considering the points in https://github.com/gardener/gardener/pull/7316, this extension should not make use of the containerd imports feature. Otherwise it might overwrite important containerd configuration.

[dimitar-kostadinov]

All sub-tasks are completed.

/close

[gardener-prow[bot]]

@dimitar-kostadinov: Closing this issue.

In response to [this](https://github.com/gardener/gardener-extension-registry-cache/issues/3#issuecomment-2147660121): >All sub-tasks are completed. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.