Error 403 from https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml

tsgsh commented 5 months ago

What happened:

Error 403 in dnf using kubernetes yum repository according to instructions at https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

What you expected to happen:

no error should occur

How to reproduce it (as minimally and precisely as possible):

[root@alice ~]# cat /etc/dnf/dnf.conf
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=True
skip_if_unavailable=False
#proxy=http://localhost:8080
[root@alice ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/1.30/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/1.30/rpm/repodata/repomd.xml.key
[root@alice ~]# dnf -y install kubectl
Kubernetes                                                                                                           300  B/s | 255  B     00:00    
Errors during downloading metadata for repository 'kubernetes':
  - Status code: 403 for https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml (IP: 18.164.68.19)
Error: Failed to download metadata for repo 'kubernetes': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
[root@alice ~]# rm -f /etc/dnf/dnf.conf.backup ; mv /etc/dnf/dnf.conf /etc/dnf/dnf.conf.backup && sed 's/^#proxy=/proxy=/' /etc/dnf/dnf.conf.backup  > /etc/dnf/dnf.conf
[root@alice ~]# sudo -u steve /usr/local/bin/mitmdump &
[1] 2024
[root@alice ~]# [19:16:51.174] HTTP(S) proxy listening at *:8080.

[root@alice ~]# dnf -y install kubectl
[19:16:59.523][[::1]:59080] client connect                  [===                                                   ] ---  B/s |   0  B     --:-- ETA
[19:16:59.541][[::1]:59080] server connect pkgs.k8s.io:443 (34.107.204.206:443)
[::1]:59080: GET https://pkgs.k8s.io/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0
 << HTTP/2.0 302 Found 138b
[19:16:59.716][[::1]:59086] client connect
[19:16:59.734][[::1]:59086] server connect prod-cdn.packages.k8s.io:443 (18.164.68.15:443)
[::1]:59086: GET https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0
 << HTTP/2.0 403 Forbidden 255b
[::1]:59080: GET https://pkgs.k8s.io/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0                           ] ---  B/s |   0  B     --:-- ETA
 << HTTP/2.0 302 Found 138b
[::1]:59086: GET https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0
 << HTTP/2.0 403 Forbidden 255b
[::1]:59080: GET https://pkgs.k8s.io/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0
 << HTTP/2.0 302 Found 138b
[::1]:59086: GET https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0
 << HTTP/2.0 403 Forbidden 255b
[::1]:59080: GET https://pkgs.k8s.io/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0                           ] 755  B/s | 255  B     --:-- ETA
 << HTTP/2.0 302 Found 138b
[::1]:59086: GET https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0
 << HTTP/2.0 403 Forbidden 255b
[19:17:00.396][[::1]:59080] client disconnect
[19:17:00.397][[::1]:59080] server disconnect pkgs.k8s.io:443 (34.107.204.206:443)
Kubernetes                                                                                                           292  B/s | 255  B     00:00    
[19:17:00.397][[::1]:59086] client disconnect
Errors during downloading metadata for repository 'kubernetes':
  - Status code: 403 for https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml (IP: ::1)
[19:17:00.398][[::1]:59086] server disconnect prod-cdn.packages.k8s.io:443 (18.164.68.15:443)
Error: Failed to download metadata for repo 'kubernetes': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
[root@alice ~]# kill 2024
[root@alice ~]#

Anything else we need to know?:

The error occurs 4 times, each from https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml (from mitmdump; see above)

Environment:

KVM VM 4GB RAM 2vCPU (AMD Ryzen 5 3600)

OS:


NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9" ALMALINUX_MANTISBT_PROJECT_VERSION="9.3" REDHAT_SUPPORT_PRODUCT="AlmaLinux" REDHAT_SUPPORT_PRODUCT_VERSION="9.3"


- Kernel (e.g. `uname -a`): `Linux alice.purplehayes.uk 5.14.0-362.24.2.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Mar 30 14:11:54 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux`
- Others:  This is a fresh kickstart install (apart from the addition of mitm to `/usr/local/bin` and the addition of a `#proxy=` line to `/etc/dnf/dnf.conf`

saschagrunert commented 5 months ago

@tsgsh thank you for the report! This feels like a temporarily issue, or is it reproducible on your side?

tsgsh commented 5 months ago

@saschagrunert thanks. It's still reproducible this morning.

tsgsh commented 5 months ago

In case it's useful, here's the details of the requests and responses gathered by mitm, indicating that it's Cloudfront that's generating the 403.

Initial request to pkgs.k8s.io

2024-04-25 13:38:29 GET https://pkgs.k8s.io/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0                                                           
                        ← 302 text/html 138b 116ms
user-agent:     libdnf (AlmaLinux 9.3; generic; Linux.x86_64)                                                                                              
accept:         */*                                                                                                                                        
cache-control:  no-cache                                                                                                                                   
pragma:         no-cache                                                                                                                                   
No request content                                                                                                                                   [m:auto]

302 Response

2024-04-25 13:38:29 GET https://pkgs.k8s.io/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0                                                           
                        ← 302 text/html 138b 116ms
server:          nginx                                                                                                                                     
date:            Thu, 25 Apr 2024 12:38:29 GMT                                                                                                             
content-type:    text/html                                                                                                                                 
content-length:  138                                                                                                                                       
location:        https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml                                 
via:             1.1 google                                                                                                                                
alt-svc:         h3=":443"; ma=2592000,h3-29=":443"; ma=2592000                                                                                            
HTML                                                                                                                                                 [m:auto]
<html>
<head>
  <title>302 Found</title>
</head>
<body>
  <center>
    <h1>302 Found</h1>
  </center>
  <hr>
  <center>nginx</center>
</body>
</html>

Redirected request to prod-cdn.packages.k8s.io

2024-04-25 13:38:30 GET https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0                
                        ← 403 application/xml 243b 120ms
                       Request                                             Response                                              Detail
user-agent:     libdnf (AlmaLinux 9.3; generic; Linux.x86_64)                                                                                              
accept:         */*                                                                                                                                        
cache-control:  no-cache                                                                                                                                   
pragma:         no-cache                                                                                                                                   
No request content                                                                                                                                   [m:auto]

403 response

2024-04-25 13:38:30 GET https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.30/rpm/repodata/repomd.xml HTTP/2.0                
                        ← 403 application/xml 243b 120ms
content-type:  application/xml                                                                                                                             
date:          Thu, 25 Apr 2024 12:38:29 GMT                                                                                                               
server:        AmazonS3                                                                                                                                    
x-cache:       Error from cloudfront                                                                                                                       
via:           1.1 8671c9c28d4abb06df55e1091d0f124a.cloudfront.net (CloudFront)                                                                            
x-amz-cf-pop:  LHR50-P4                                                                                                                                    
x-amz-cf-id:   FoL2-AO-RG5TZX_kpSqtSop2j7_dCvpyar1LDwR4jpLBsvMT797sqA==                                                                                    
XML                                                                                                                                                  [m:auto]
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AccessDenied</Code>
  <Message>Access Denied</Message>
  <RequestId>40AF0019CBBVAH8T</RequestId>
  <HostId>OiMpm2o+2j3cVp+EJfh+AK0ZTUe3oNxZ6VihktmT2hNPZTh9vnNXKAgAGK3L8mW8JAaBWHkDS/c=</HostId>
</Error>

xmudrii commented 5 months ago

@tsgsh You have a mistake in your URLs, it should be v1.30, not 1.30

tsgsh commented 5 months ago

@xmudrii thanks. I'd checked that thoroughly several times and didn't see it! It's a mistake in a Jinja2 template expression that creates that.

Sorry for the trouble!

kubernetes / release