Error installing from official repos onto Ubuntu

[ganeshgunasekaran]

Hi,

I am tried installing the kubeadm in Ubuntu 22.4 LTS following the instructions given in the page https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ I was able follow the steps and successfully installed Containerd . The commands worked fine till Installing kubeadm -> Debian-based distributions -> Add the appropriate Kubernetes apt repository The next step showed error while running "sudo apt-get update"

Please find the command outputs attached as screenshots Step 1-2 and step 3-4.

This if my first issue. Please correct me if I have should have done something else.

[k8s-ci-robot]

This issue is currently awaiting triage.

SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[sftim]

Thanks for reporting this /retitle Error (missing repository signature) installing from official repos onto Ubuntu /kind bug

/sig release (and specifically @xmudrii) might like to know about this

[sftim]

@ganeshgunasekaran, what happens when you run this: curl --verbose -i -L https://pkgs.k8s.io/core:/stable:/v1.28/deb/InRelease

The error output, if there is one, might help you understand what to fix.

[ganeshgunasekaran]

Thanks a lot for picking up this issue. @sftim . Please find the output of the command pasted below

ubuntu1@ubuntu1:~$ cat curl_run.sh curl --verbose -i -L https://pkgs.k8s.io/core:/stable:/v1.28/deb/ ubuntu1@ubuntu1:~$ ./curl_run.sh > curl_run.out 2>&1 ubuntu1@ubuntu1:~$ cat curl_run.out % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 34.107.204.206:443...

• Connected to pkgs.k8s.io (34.107.204.206) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• TLSv1.0 (OUT), TLS header, Certificate Status (22): } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data]
• TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data]
• TLSv1.2 (IN), TLS header, Finished (20): { [5 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [15 bytes data]
• TLSv1.3 (IN), TLS handshake, Certificate (11): { [5302 bytes data]
• TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data]
• TLSv1.3 (IN), TLS handshake, Finished (20): { [52 bytes data]
• TLSv1.2 (OUT), TLS header, Finished (20): } [5 bytes data]
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data]
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Finished (20): } [52 bytes data]
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
• ALPN, server accepted to use h2
• Server certificate:
• subject: CN=k8s.io
• start date: Jul 27 22:56:09 2023 GMT
• expire date: Oct 25 23:32:07 2023 GMT
• subjectAltName: host "pkgs.k8s.io" matched cert's "pkgs.k8s.io"
• issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
• SSL certificate verify ok.
• Using HTTP2, server supports multiplexing
• Connection state changed (HTTP/2 confirmed)
• Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]
• Using Stream ID: 1 (easy handle 0x5602ae8bbe90)
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]

  GET /core:/stable:/v1.28/deb/ HTTP/2 Host: pkgs.k8s.io user-agent: curl/7.81.0 accept: /

• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [267 bytes data]
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [267 bytes data]
• old SSL session ID is stale, removing
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data] < HTTP/2 302 < server: nginx < date: Sat, 19 Aug 2023 10:26:18 GMT < content-type: text/html < content-length: 138 < location: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/ < via: 1.1 google < alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 <
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• Ignoring the response-body { [138 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data] 100 138 100 138 0 0 363 0 --:--:-- --:--:-- --:--:-- 365
• Connection #0 to host pkgs.k8s.io left intact
• Issue another request to this URL: 'https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/'
• Trying 13.224.181.102:443...
• Connected to prod-cdn.packages.k8s.io (13.224.181.102) port 443 (#1)
• ALPN, offering h2
• ALPN, offering http/1.1
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• TLSv1.0 (OUT), TLS header, Certificate Status (22): } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data]
• TLSv1.2 (IN), TLS header, Certificate Status (22): { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data]
• TLSv1.2 (IN), TLS header, Finished (20): { [5 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [19 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Certificate (11): { [4972 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data]
• TLSv1.2 (OUT), TLS header, Finished (20): } [5 bytes data]
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data]
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data]
• SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
• ALPN, server accepted to use h2
• Server certificate:
• subject: CN=prod-cdn.packages.k8s.io
• start date: Jul 12 00:00:00 2023 GMT
• expire date: Aug 9 23:59:59 2024 GMT
• subjectAltName: host "prod-cdn.packages.k8s.io" matched cert's "prod-cdn.packages.k8s.io"
• issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
• SSL certificate verify ok.
• Using HTTP2, server supports multiplexing
• Connection state changed (HTTP/2 confirmed)
• Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]
• Using Stream ID: 1 (easy handle 0x5602ae8bbe90)
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]

  GET /repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/ HTTP/2 Host: prod-cdn.packages.k8s.io user-agent: curl/7.81.0 accept: /

• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [124 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data]
• Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
• TLSv1.2 (OUT), TLS header, Supplemental data (23): } [5 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data] 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0* TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data] < HTTP/2 403 < content-type: application/xml < date: Sat, 19 Aug 2023 10:26:19 GMT < server: AmazonS3 < x-cache: Error from cloudfront < via: 1.1 291933b5bb7fbb03efd999a83bb9696a.cloudfront.net (CloudFront) < x-amz-cf-pop: SYD1-C2 < x-amz-cf-id: uQ8T7zHTABIpTKNf3DqSYdISb8s1W96gMF5itC632lLxKQ0chyl3uQ== < { [255 bytes data]
• TLSv1.2 (IN), TLS header, Supplemental data (23): { [5 bytes data] 100 255 0 255 0 0 126 0 --:--:-- 0:00:02 --:--:-- 2040
• Connection kubernetes/website#1 to host prod-cdn.packages.k8s.io left intact HTTP/2 302 server: nginx date: Sat, 19 Aug 2023 10:26:18 GMT content-type: text/html content-length: 138 location: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/ via: 1.1 google alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

HTTP/2 403 content-type: application/xml date: Sat, 19 Aug 2023 10:26:19 GMT server: AmazonS3 x-cache: Error from cloudfront via: 1.1 291933b5bb7fbb03efd999a83bb9696a.cloudfront.net (CloudFront) x-amz-cf-pop: SYD1-C2 x-amz-cf-id: uQ8T7zHTABIpTKNf3DqSYdISb8s1W96gMF5itC632lLxKQ0chyl3uQ==

<?xml version="1.0" encoding="UTF-8"?>

AccessDeniedAccess DeniedXB235ZQQ726WQQHVBV9V/HkvxgZtzhn+aRTfpP3Yh03zGoj3NEbm/dboQddDwSSXW1WTHX4GApdKt50YgIB/jFL2IaP1dMIBAsgQ6g==ubuntu1@ubuntu1:~$

[sftim]

Looks like the docs are correct.

/transfer kubernetes

[sftim]

/retitle Error (403 Forbidden) installing from official repos onto Ubuntu /triage needs-information

[sftim]

@ganeshgunasekaran what happens when you run this exact command: curl --verbose -i -L https://pkgs.k8s.io/core:/stable:/v1.28/deb/InRelease

?

[ganeshgunasekaran]

Hi @sftim , Sorry about the mistake. Please find the response below. ubuntu1@ubuntu1:~$ curl --verbose -i -L https://pkgs.k8s.io/core:/stable:/v1.28/deb/InRelease

• Trying 34.107.204.206:443...
• Connected to pkgs.k8s.io (34.107.204.206) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• TLSv1.0 (OUT), TLS header, Certificate Status (22):
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• TLSv1.2 (IN), TLS header, Certificate Status (22):
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS header, Finished (20):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
• TLSv1.3 (IN), TLS handshake, Certificate (11):
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
• TLSv1.3 (IN), TLS handshake, Finished (20):
• TLSv1.2 (OUT), TLS header, Finished (20):
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• TLSv1.3 (OUT), TLS handshake, Finished (20):
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
• ALPN, server accepted to use h2
• Server certificate:
• subject: CN=k8s.io
• start date: Jul 27 22:56:09 2023 GMT
• expire date: Oct 25 23:32:07 2023 GMT
• subjectAltName: host "pkgs.k8s.io" matched cert's "pkgs.k8s.io"
• issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
• SSL certificate verify ok.
• Using HTTP2, server supports multiplexing
• Connection state changed (HTTP/2 confirmed)
• Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• Using Stream ID: 1 (easy handle 0x560f5e3b0e90)
• TLSv1.2 (OUT), TLS header, Supplemental data (23):

  GET /core:/stable:/v1.28/deb/InRelease HTTP/2 Host: pkgs.k8s.io user-agent: curl/7.81.0 accept: /

• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• old SSL session ID is stale, removing
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.2 (IN), TLS header, Supplemental data (23): < HTTP/2 302 HTTP/2 302 < server: nginx server: nginx < date: Mon, 21 Aug 2023 07:56:26 GMT date: Mon, 21 Aug 2023 07:56:26 GMT < content-type: text/html content-type: text/html < content-length: 138 content-length: 138 < location: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/InRelease location: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/InRelease < via: 1.1 google via: 1.1 google < alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

<

• TLSv1.2 (IN), TLS header, Supplemental data (23):
• Ignoring the response-body
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• Connection #0 to host pkgs.k8s.io left intact
• Issue another request to this URL: 'https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/InRelease'
• Trying 13.224.181.114:443...
• Connected to prod-cdn.packages.k8s.io (13.224.181.114) port 443 (#1)
• ALPN, offering h2
• ALPN, offering http/1.1
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• TLSv1.0 (OUT), TLS header, Certificate Status (22):
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• TLSv1.2 (IN), TLS header, Certificate Status (22):
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS header, Finished (20):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.3 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.3 (IN), TLS handshake, Finished (20):
• TLSv1.2 (OUT), TLS header, Finished (20):
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• TLSv1.3 (OUT), TLS handshake, Finished (20):
• SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
• ALPN, server accepted to use h2
• Server certificate:
• subject: CN=prod-cdn.packages.k8s.io
• start date: Jul 12 00:00:00 2023 GMT
• expire date: Aug 9 23:59:59 2024 GMT
• subjectAltName: host "prod-cdn.packages.k8s.io" matched cert's "prod-cdn.packages.k8s.io"
• issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
• SSL certificate verify ok.
• Using HTTP2, server supports multiplexing
• Connection state changed (HTTP/2 confirmed)
• Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• Using Stream ID: 1 (easy handle 0x560f5e3b0e90)
• TLSv1.2 (OUT), TLS header, Supplemental data (23):

  GET /repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/InRelease HTTP/2 Host: prod-cdn.packages.k8s.io user-agent: curl/7.81.0 accept: /

• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
• TLSv1.2 (OUT), TLS header, Supplemental data (23):
• TLSv1.2 (IN), TLS header, Supplemental data (23):
• TLSv1.2 (IN), TLS header, Supplemental data (23): < HTTP/2 200 HTTP/2 200 < content-type: application/octet-stream content-type: application/octet-stream < content-length: 1186 content-length: 1186 < last-modified: Tue, 15 Aug 2023 17:06:48 GMT last-modified: Tue, 15 Aug 2023 17:06:48 GMT < x-amz-server-side-encryption: AES256 x-amz-server-side-encryption: AES256 < x-amz-meta-mtime: 1692119134.983891584 x-amz-meta-mtime: 1692119134.983891584 < accept-ranges: bytes accept-ranges: bytes < server: AmazonS3 server: AmazonS3 < date: Mon, 21 Aug 2023 02:06:07 GMT date: Mon, 21 Aug 2023 02:06:07 GMT < etag: "63d4fc87d6c0e8739c65f55f4d8aa600" etag: "63d4fc87d6c0e8739c65f55f4d8aa600" < vary: Accept-Encoding vary: Accept-Encoding < x-cache: Hit from cloudfront x-cache: Hit from cloudfront < via: 1.1 41f4e34e5d78c923aead0fa16ff91eb8.cloudfront.net (CloudFront) via: 1.1 41f4e34e5d78c923aead0fa16ff91eb8.cloudfront.net (CloudFront) < x-amz-cf-pop: SYD1-C2 x-amz-cf-pop: SYD1-C2 < x-amz-cf-id: jjbla7ya0KBxnHn2fnwxqV8vxcHGlaM-08_ZXBfnCFDLDslruglVwg== x-amz-cf-id: jjbla7ya0KBxnHn2fnwxqV8vxcHGlaM-08_ZXBfnCFDLDslruglVwg== < age: 21574 age: 21574

< -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

Archive: deb Codename: deb Origin: obs://build.opensuse.org/isv:kubernetes:core:stable:v1.28/deb Label: isv:kubernetes:core:stable:v1.28 Architectures: amd64 arm64 s390x ppc64el Date: Tue Aug 15 17:05:34 2023 Description: Kubernetes v1.28 (Stable) (deb) MD5Sum: 61f5b9d38a31b1f3213816c8dfb3e85a 11872 Packages 5804a31770caf87fbd6beb6c5c53920b 2759 Packages.gz SHA1: 3245f8ed9874d24198d54f94bb5eca770f8cc11a 11872 Packages e13e2873d2d11928b642410f7bd28db092aa7796 2759 Packages.gz SHA256: a8ec729af2342f13728bcd0e93b9dc3e512025972f6a8a3778f6cf8bd12c6c40 11872 Packages 0a9f6e3a6f234d021c1098a59f326f1abedc31f63b8d297c738a98cc4413bc8c 2759 Packages.gz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux)

iQEVAwUBZNuwXiNGVNqaKWQ2AQiSRQf/V/9DWBbLR54XkMrtohORmPmrVdcBRTlS LzzTjxC0jlwznQf4GpLwnruKdseruohFJZ8obEF9+7bst55gbRISs5iqa92UfduB sM6K8tTpw9HLOvQbBTjYcn7G7LgF7qtbJHFRoPteCbZDcfQsFeCrw+3c3T4Vt2+A XE2seMnvUgJFk8Pj5bD7182OTeK+wuH7j8F6VPhY+DWUTG8mtKPuRL3nfZX7HA1p 6Nxo9NQSdGRso8han1ocJkhUv90AYBfw3vKIZQRApKcvANIZvcVi0Iyd2q7saztg j5ly3NRwPpGsu+ObDbdknzYpcXm+Prmg9Z1qX4L/wg3kkg7yHRUI1w== =DV5/ -----END PGP SIGNATURE-----

• TLSv1.2 (IN), TLS header, Supplemental data (23):
• Connection kubernetes/kubernetes#1 to host prod-cdn.packages.k8s.io left intact ubuntu1@ubuntu1:~$

[sftim]

/retitle Error installing from official repos onto Ubuntu

Feels like a support query; I'm not sure /remove-triage needs-information

(for now)

[sftim]

/retitle Error installing from official repos onto Ubuntu

[xmudrii]

/transfer release /assign I'll be taking a look into this issue.

[xmudrii]

/triage accepted

[dominic-p]

For what it's worth, I'm also observing the "Access Denied" issue above. I'm not very familiar with how an APT repo is supposed to work, but most of the ones I have used allow browsing available packages (e.g. https://apt.kubernetes.io/).

[xmudrii]

@dominic-p What cloud provider or infrastructure are you using? For example, AWS tends to block access from some Hetzner IP addresses and we're aware of that, but there's nothing that we can do about that. Speaking of browsing the repo, this is not supported via pkgs.k8s.io, but you can do that via download.opensuse.org: https://download.opensuse.org/repositories/isv:/kubernetes:/

[dominic-p]

Thanks for the quick reply. That is very interesting as I am using Hetzner, and I've been struggling with network issues in a lot of areas. I assumed it was a misconfiguration on my end, but maybe my IPs are blacklisted.

Just so I'm clear, it is expected behavior to get "Access Denied" when visiting https://pkgs.k8s.io/core:/stable:/v1.28/deb/ correct?

I originally came here because after switching from the Google repos to the community ones, my Hetzner load balancers no longer get any targets. The exact same config/version works when I install from the Google repos. I can open an issue on Hetzner's CCM, but I thought I would check here first. Are there any differences in the actual packages between the Google repos and the community ones?

[xmudrii]

Just so I'm clear, it is expected behavior to get "Access Denied" when visiting https://pkgs.k8s.io/core:/stable:/v1.28/deb/ correct?

Yes, it's expected. We don't have a file browser at pkgs.k8s.io.

Are there any differences in the actual packages between the Google repos and the community ones?

There should be no major differences. For both the Google repos and the community repos, we use the same binaries (e.g. kubelet, kubeadm...). Have you checked kubelet logs to make sure that kubelet is installed and running?

[dominic-p]

Thanks for the confirmation. Yes, the kubelet is running. I checked the logs, and I didn't see anything that looked out of the ordinary. It seems to have trouble connecting to the CRI-O unix socket for a bit at startup and then everything looks good. I'll take a look at the .deb files from each repo to see if I can see any differences.

[xmudrii]

@dominic-p Sounds good and please let us know if you find any difference. We tried our best to match these debs, and if there's any difference, it would be good to address it if it's possible.

[dominic-p]

Ok, I downloaded 1.28 kubelet debs from both the google repo and the community repo just now using a fresh Debian v12 container running on my cluster. I haven't looked deeply into the packages yet, but there are definitely some differences in the packages I downloaded.

$ tree
.
├── community
│   ├── etc
│   │   ├── kubernetes
│   │   │   └── manifests
│   │   └── sysconfig
│   │       └── kubelet
│   ├── lib
│   │   └── systemd
│   │       └── system
│   │           └── kubelet.service
│   ├── usr
│   │   ├── bin
│   │   │   └── kubelet
│   │   └── share
│   │       └── doc
│   │           └── kubelet
│   │               ├── LICENSE
│   │               └── README.md
│   └── var
│       └── lib
│           └── kubelet
└── google
    ├── lib
    │   └── systemd
    │       └── system
    │           └── kubelet.service
    └── usr
        └── bin
            └── kubelet

Here are the steps I did to get the debs:

Community

1. Create new Debian 12 container
2. Run apt update followed by apt install curl gpg
3. Add community apt repo:

   
   curl -fsSL "https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key" | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list

4. Download kubelet: `apt download kubelet`: kubelet_1.28.1-1.1_amd64.deb

### Google

1. Create new Debian 12 container
2. Run `apt update` followed by `apt install curl gpg`
3. Add Google apt repo: 

curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list

4. Download kubelet: `apt download "kubelet=1.28.*"`: kubelet_1.28.1-00_amd64.deb

[xmudrii]

@dominic-p The new packages have additional files that we didn't have in packages published to the legacy Google-hosted repositories. It's important that we don't have missing files, i.e. that we installed some file with old packages, but that we don't do it with new packages.

[dominic-p]

Thanks for the explanation there. Ok, I think I was able to find the issue. The Hetzner CCM currently requires --cloud-provider=external to be set in KUBELET_EXTRA_ARGS. My configuration script sets the env variable in /etc/default/kubelet and the community package includes a new file /etc/sysconfig/kubelet with the contents KUBELET_EXTRA_ARGS=. That resets the env variable and breaks the Hetzner CCM.

I guess that makes this a bug with my particular configuration (I've already worked around it), but it is strange to me that the env variable is set in the new deb package when it wasn't before.

[xmudrii]

There's another report relevant to /etc/default/kubelet: https://github.com/kubernetes/release/issues/3276 We'll be taking a look into this as soon as possible.

[AnirudhPanchangam]

Hi Team,

I am experiencing a similar issue. Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease [1,186 B] Err:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436 Reading package lists... Done W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.28/deb InRelease' is not signed. N: Updating from such a repository can't be done securely, and is therefore disabled by default.

I've run the same commands that OP has run. However, i get NO_PUBKEY issue. I tried running sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 234654DA9A296436 Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). Executing: /tmp/apt-key-gpghome.1PwUILV6CQ/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys 234654DA9A296436 gpg: key 234654DA9A296436: public key "isv:kubernetes OBS Project isv:kubernetes@build.opensuse.org" imported gpg: Total number processed: 1 gpg: imported: 1

However, i still get the same error upon running apt-get update This is an error that i have started getting recently. It used to work just fine up until yesterday.

Please let me know if there is anything else i can try.

Thanks, Anirudh

[nethershaw]

I'm seeing this 403 response both from within AWS on EC2 instances and from my own system at home on a standard cable ISP. Same error from CloudFront.

I can see it's just CloudFront -> S3. Consider checking your bucket policy, and remember that S3 returns HTTP 403 not just for access denied, but also when it would return HTTP 404. It does this on purpose so that unauthenticated users cannot use return codes to tell what files exist in the bucket. If you are looking for a permissions/policy problem, but it is actually a pathing problem, this behavior will conceal it.

Given the above... it seems the entire problem is this:

• https://pkgs.k8s.io/core:/stable:/v1.28/deb/InRelease exists.
• https://pkgs.k8s.io/core:/stable:/v1.28/deb/dists/jammy/InRelease does not exist.
• Apt assumes the latter should exist when given https://pkgs.k8s.io/core:/stable:/v1.28/deb.

[xmudrii]

@AnirudhPanchangam @nethershaw At this time, please use the official instructions for adding the repository (e.g. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl).

[lucasmo]

Hi Team,

I am experiencing a similar issue. Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease [1,186 B] Err:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436 Reading package lists... Done W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.28/deb InRelease' is not signed. N: Updating from such a repository can't be done securely, and is therefore disabled by default.

I've run the same commands that OP has run. However, i get NO_PUBKEY issue.

Are you running Ubuntu 20.04 by chance? I had to run these commands and now the NO_PUBKEY issue went away:

sudo chmod 755 /etc/apt/keyrings
sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg

A failure like "can't read signed-by key" or something would have been more helpful, apt.

[xmudrii]

Hey folks!

This issue now contains multiple distinct issues:

• 403 Forbidden trying to browse files: we don't have a file browser and the underlying S3 bucket is private, so accessing URLs of directories such as https://pkgs.k8s.io/core:/stable:/v1.28/deb is not going to work. #3317 has been created to track this
• 403 Forbidden trying to install packages: some cloud providers have IP addresses blocked by AWS. Hetzner is often affected by this. There is nothing that we can do about this, the same issue is affecting registry.k8s.io, and we can't make AWS to unblock the affected IP addresses. You can only try to get a new IP address that's not blocked or to mirror the repository somewhere else
• The issue with /etc/default/kubelet reported by @dominic-p is already fixed as part of #3279
• For issues like can't read signed-by key, we strongly recommend setting up repositories as described in the official guidelines: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl
• About Apt assumes the latter should exist when given https://pkgs.k8s.io/core:/stable:/v1.28/deb reported by @nethershaw: we couldn't reproduce this issue, please try setting up the repository as described in the document that I linked previously. If the issue still appears, please create a new issue in this repository

Given that this issue contains multiple different reports and is hard to navigate, I'll go ahead and lock it. If you run into any issue and it's not already covered by this issue/comment, please create a new issue in this repository.