search

berops / claudie

Cloud-agnostic managed Kubernetes
https://docs.claudie.io/
Apache License 2.0
526 stars 34 forks source link

Bug: 403 Forbidden when pulling pause containers #1272

Open cloudziu opened 4 months ago

cloudziu commented 4 months ago

This issue is related to: https://github.com/kubernetes/registry.k8s.io/issues/261 and https://github.com/berops/claudie/issues/783

In a cluster deployed on Hetzner, after a cluster is deployed one of the compute nodes stays in NotReady state. From the node logs we can see:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "registry.k8s.io/pause:3.9": failed to pull image "registry.k8s.io/pause:3.9": failed to pull and unpack image "registry.k8s.io/pause:3.9": failed to resolve reference "registry.k8s.io/pause:3.9": unexpected status from HEAD request to https://europe-west3-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9: 403 Forbidden

I've tried to manually pull the images from the VM with and the result is the same 403 Forbidden:

# ctr --debug image pull --http-dump --http-trace -k registry.k8s.io/pause:3.9
DEBU[0000] fetching                                      image="registry.k8s.io/pause:3.9"
DEBU[0000] resolving                                     host=registry.k8s.io
DEBU[0000] do request                                    host=registry.k8s.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.28 request.method=HEAD url="https://registry.k8s.io/v2/pause/manifests/3.9"
INFO[0000] HEAD /v2/pause/manifests/3.9 HTTP/1.1
INFO[0000] Host: registry.k8s.io
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*
INFO[0000] User-Agent: containerd/1.6.28
INFO[0000]
DEBU[0000] DNS lookup                                    host=registry.k8s.io
DEBU[0000] DNS lookup complete                           coalesced=false result=34.96.108.209
DEBU[0000] Connection successful                         remote_addr="34.96.108.209:443" reused=false
INFO[0000] HTTP/1.1 307 Temporary Redirect
INFO[0000] Transfer-Encoding: chunked
INFO[0000] Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
INFO[0000] Content-Type: text/html; charset=utf-8
INFO[0000] Date: Wed, 13 Mar 2024 13:02:46 GMT
INFO[0000] Location: https://europe-west3-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9
INFO[0000] Server: Google Frontend
INFO[0000] Via: 1.1 google, 1.1 google
INFO[0000] X-Cloud-Trace-Context: 7c0c3ef8fc6400fae1f36fdb1d0e223e
INFO[0000]
INFO[0000]
INFO[0000] HEAD /v2/k8s-artifacts-prod/images/pause/manifests/3.9 HTTP/0.0
INFO[0000] Host: europe-west3-docker.pkg.dev
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*
INFO[0000] Referer: https://registry.k8s.io/v2/pause/manifests/3.9
INFO[0000] User-Agent: containerd/1.6.28
INFO[0000]
DEBU[0000] DNS lookup                                    host=europe-west3-docker.pkg.dev
DEBU[0000] DNS lookup complete                           coalesced=false result=173.194.76.82
DEBU[0000] Connection successful                         remote_addr="173.194.76.82:443" reused=false
INFO[0000] HTTP/1.1 403 Forbidden
INFO[0000] Transfer-Encoding: chunked
INFO[0000] Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
INFO[0000] Content-Type: text/html; charset=UTF-8
INFO[0000] Date: Wed, 13 Mar 2024 13:02:46 GMT
INFO[0000]
INFO[0000]
DEBU[0000] fetch response received                       host=registry.k8s.io response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000" response.header.content-type="text/html; charset=UTF-8" response.header.date="Wed, 13 Mar 2024 13:02:46 GMT" response.status="403 Forbidden" url="https://registry.k8s.io/v2/pause/manifests/3.9"
ctr: failed to resolve reference "registry.k8s.io/pause:3.9": unexpected status from HEAD request to https://europe-west3-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9: 403 Forbidden

How it differs from #783 is that the cluster was able do provision and spinup a node.

Despire commented 2 months ago

I wasn't able to setup a HTTPS proxy, spend 2 days.

To use the Proxy it needs to handle HTTPS traffic on port 443 but also on 6443. it also needs to handle HTTP traffic (which was the easy step that worked)

During the HTTPS proxy setup I stumbled upon random issues like 502 badgateway, SSL CONNECT errors or some TLS1.3 errors

You will also need a valid certificate I used the one for claudie.dev on cloudflare

I tried to setup the kubeone HTTP proxy e.g. https://docs.kubermatic.com/kubeone/v1.7/guides/proxy/