search

kubernetes / registry.k8s.io

This project is the repo for registry.k8s.io, the production OCI registry service for Kubernetes' container image artifacts
https://registry.k8s.io
Apache License 2.0
397 stars 69 forks source link

Cannot pull images from AWS EKS Cluster (Region eu-central1, Frankfurt) #245

Closed sebschlue closed 1 year ago

sebschlue commented 1 year ago

Is there an existing issue for this?

What did you expect to happen?

Image pull from registry.k8s.io should work but I simply get 403 Forbidden

Debugging Information

Normal Pulling 87s (x4 over 2m58s) kubelet Pulling image "registry.k8s.io/metrics-server/metrics-server:v0.6.3" Warning Failed 86s (x4 over 2m57s) kubelet Failed to pull image "registry.k8s.io/metrics-server/metrics-server:v0.6.3": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/metrics-server/metrics-server:v0.6.3": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry.k8s.io/v2/metrics-server/metrics-server/blobs/sha256:817bbe3f2e5179b5fa7dd4085c87aab1faaf9dc879e3b1d05796901a006f58ab: 403 Forbidden

Anything else?

No response

Code of Conduct

ameukam commented 1 year ago

@sebschlue Do you mind go through the debugging guide so we can get more informations about this issue you are facing ?

What you provided is enough for us to properly investigate this issue.

sebschlue commented 1 year ago

I will try some tests.

First test:


$ kubectl run curl --rm -it --image=curlimages/curl --restart=Never --command -- curl -v https://registry.k8s.io/v2/
*   Trying 34.96.108.209:443...
* Connected to registry.k8s.io (34.96.108.209) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /cacert.pem
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=registry.k8s.io
*  start date: May  9 06:25:11 2023 GMT
*  expire date: Aug  7 07:21:05 2023 GMT
*  subjectAltName: host "registry.k8s.io" matched cert's "registry.k8s.io"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /v2/]
* h2h3 [:scheme: https]
* h2h3 [:authority: registry.k8s.io]
* h2h3 [user-agent: curl/8.0.1-DEV]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7ff772e90a90)
> GET /v2/ HTTP/2
> Host: registry.k8s.io
> user-agent: curl/8.0.1-DEV
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
< docker-distribution-api-version: registry/2.0
< x-cloud-trace-context: 928d497daa6bd00cb18f27f7dfc768c9
< date: Thu, 11 May 2023 09:23:38 GMT
< content-type: text/html
< server: Google Frontend
< content-length: 0
< via: 1.1 google
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<
* Connection #0 to host registry.k8s.io left intact
pod "curl" deleted````
sebschlue commented 1 year ago

Second test:

$ kubectl run crane --rm -it --image=gcr.io/go-containerregistry/crane --restart=Never --command -- crane ls registry.k8s.io/metrics-server
dev
v0.2.1
v0.3.3
v0.3.4
v0.3.5
v0.3.6
pod "crane" deleted
sebschlue commented 1 year ago

third test:


$ kubectl run crane --rm -it --image=gcr.io/go-containerregistry/crane --restart=Never --command -- crane pull --verbose registry.k8s.io/metrics-server:v0.3.6 metrics-server-v0.3.6.tgz
If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/crane, falling back to streaming logs:
2023/05/11 09:46:49 --> GET https://registry.k8s.io/v2/
2023/05/11 09:46:49 GET /v2/ HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept-Encoding: gzip

2023/05/11 09:46:49 <-- 200 https://registry.k8s.io/v2/ (79.136354ms)
2023/05/11 09:46:49 HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html
Date: Thu, 11 May 2023 09:46:49 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: aed4ad49eb58ccf06be191eafb4e8199
Content-Length: 0

2023/05/11 09:46:49 --> GET https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6
2023/05/11 09:46:49 GET /v2/metrics-server/manifests/v0.3.6 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Accept-Encoding: gzip

2023/05/11 09:46:49 <-- 307 https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6 (20.375601ms)
2023/05/11 09:46:49 HTTP/2.0 307 Temporary Redirect
Content-Length: 132
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 11 May 2023 09:46:49 GMT
Location: https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: fff6db6c3bb348da31246ec53538b46a

<a href="https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6">Temporary Redirect</a>.

2023/05/11 09:46:49 --> GET https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6
2023/05/11 09:46:49 GET /v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6 HTTP/1.1
Host: europe-west4-docker.pkg.dev
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Referer: https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6
Accept-Encoding: gzip

2023/05/11 09:46:49 <-- 200 https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6 (108.131929ms)
2023/05/11 09:46:49 HTTP/2.0 200 OK
Content-Length: 738
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Date: Thu, 11 May 2023 09:46:49 GMT
Docker-Content-Digest: sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1784,
      "digest": "sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 654467,
         "digest": "sha256:e8d8785a314f385d3675a017f4e2df1707c528c06e7a7989663fdab4900bd8ff"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 9885841,
         "digest": "sha256:b2f4b24bed0dade029d7164c181e0c8b2c0846fb3a1e2aabf4141480382849b2"
      }
   ]
}
Error: saving tarball metrics-server-v0.3.6.tgz: open metrics-server-v0.3.6.tgz: permission denied
pod "crane" deleted
pod vault/crane terminated (Error)```
TerryHowe commented 1 year ago

Last message just shows permission denied for writing the tarball. Did you get past that?

sebschlue commented 1 year ago

Yes :) Now you can see the 403 at the end:

$ kubectl run crane --rm -it --image=gcr.io/go-containerregistry/crane --restart=Never --command -- crane pull --verbose registry.k8s.io/metrics-server:v0.3.6 /tmp/metrics-server-v0.3.6.tgz
2023/05/11 15:03:21 --> GET https://registry.k8s.io/v2/
2023/05/11 15:03:21 GET /v2/ HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept-Encoding: gzip

2023/05/11 15:03:21 <-- 200 https://registry.k8s.io/v2/ (56.484713ms)
2023/05/11 15:03:21 HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html
Date: Thu, 11 May 2023 15:03:21 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 1a61c1a94b76867c3e81ec2fb1e68981
Content-Length: 0

2023/05/11 15:03:21 --> GET https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6
2023/05/11 15:03:21 GET /v2/metrics-server/manifests/v0.3.6 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Accept-Encoding: gzip

2023/05/11 15:03:21 <-- 307 https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6 (20.880947ms)
2023/05/11 15:03:21 HTTP/2.0 307 Temporary Redirect
Content-Length: 132
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 11 May 2023 15:03:21 GMT
Location: https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 67ede95221b1d2fa5bdf3816b9233c3d

<a href="https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6">Temporary Redirect</a>.

2023/05/11 15:03:21 --> GET https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6
2023/05/11 15:03:21 GET /v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6 HTTP/1.1
Host: europe-west4-docker.pkg.dev
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Referer: https://registry.k8s.io/v2/metrics-server/manifests/v0.3.6
Accept-Encoding: gzip

2023/05/11 15:03:22 <-- 200 https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/metrics-server/manifests/v0.3.6 (102.210221ms)
2023/05/11 15:03:22 HTTP/2.0 200 OK
Content-Length: 738
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Date: Thu, 11 May 2023 15:03:21 GMT
Docker-Content-Digest: sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1784,
      "digest": "sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 654467,
         "digest": "sha256:e8d8785a314f385d3675a017f4e2df1707c528c06e7a7989663fdab4900bd8ff"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 9885841,
         "digest": "sha256:b2f4b24bed0dade029d7164c181e0c8b2c0846fb3a1e2aabf4141480382849b2"
      }
   ]
}
2023/05/11 15:03:22 --> GET https://registry.k8s.io/v2/metrics-server/blobs/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b
2023/05/11 15:03:22 GET /v2/metrics-server/blobs/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Accept-Encoding: gzip

2023/05/11 15:03:22 <-- 307 https://registry.k8s.io/v2/metrics-server/blobs/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b (88.617019ms)
2023/05/11 15:03:22 HTTP/2.0 307 Temporary Redirect
Content-Length: 207
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Thu, 11 May 2023 15:03:22 GMT
Location: https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b
Server: Google Frontend
Via: 1.1 google
X-Cloud-Trace-Context: 2cd3976c07d6d9a3e676c04a710dce09

<a href="https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b">Temporary Redirect</a>.

2023/05/11 15:03:22 --> GET https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b
2023/05/11 15:03:22 GET /containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b HTTP/1.1
Host: prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com
User-Agent: crane/a927d7c995a991edeb54e840f66d8cfffcf30d62 go-containerregistry/(devel)
Referer: https://registry.k8s.io/v2/metrics-server/blobs/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b
Accept-Encoding: gzip

2023/05/11 15:03:22 <-- 403 https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b (36.755538ms)
2023/05/11 15:03:22 HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 11 May 2023 15:03:21 GMT
Server: AmazonS3
X-Amz-Id-2: Y5uEtfR7hGc+oYpqpEYA6suxba76tt6ZHC+9/VlThMMEdnqwmwyfEft3X52hSSdm8hmjLuyY8nA=
X-Amz-Request-Id: T4K2XR8208VF2NTX

f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>T4K2XR8208VF2NTX</RequestId><HostId>Y5uEtfR7hGc+oYpqpEYA6suxba76tt6ZHC+9/VlThMMEdnqwmwyfEft3X52hSSdm8hmjLuyY8nA=</HostId></Error>
0

Error: saving tarball /tmp/metrics-server-v0.3.6.tgz: unable to calculate manifest: GET https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b: unexpected status code 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>T4K2XR8208VF2NTX</RequestId><HostId>Y5uEtfR7hGc+oYpqpEYA6suxba76tt6ZHC+9/VlThMMEdnqwmwyfEft3X52hSSdm8hmjLuyY8nA=</HostId></Error>
BenTheElder commented 1 year ago

This works from my machine:

curl https://prod-registry-k8s-io-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/containers/images/sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b

So it's not the blob at least, something between S3 and your host.

Is it possible your environment has some restrictions on S3 access? Previously https://github.com/kubernetes/k8s.io/issues/4214#issuecomment-1247446149

BenTheElder commented 1 year ago

https://docs.aws.amazon.com/AmazonS3/latest/userguide/troubleshoot-403-errors.html

Perhaps https://docs.aws.amazon.com/AmazonS3/latest/userguide/troubleshoot-403-errors.html#troubleshoot-403-orgs

sebschlue commented 1 year ago

Thanks a lot, will check it :)

sebschlue commented 1 year ago

Okay, I did not yet figure out the root cause, but it's now clear that it is not a general issue but limited to my environment, so ticket can be closed. Thanks a lot for your help :)