apk update - ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: Permission denied

ronnicek commented 4 years ago

Hi,

sorry, if this bug is reported towards the wrong repo.

We hit following errors in edge:

Alpine latest:
[root@dockerserver ~]# docker run -it --rm alpine:latest /bin/sh
/ # export http_proxy=http://<proxyserver>:3128
/ # export https_proxy=http://<proxyserver>:3128
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
v3.12.0-160-g96d355a76e [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
v3.12.0-161-g4b08580d3e [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
OK: 12749 distinct packages available
/ #

Alpine Edge:
[root@dockerserver ~]# docker run -it --rm alpine:edge /bin/sh
/ # export http_proxy=http://<proxyserver>:3128
/ # export https_proxy=http://<proxyserver>:3128
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
< after milion years later >
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: Permission denied
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
< after another milion years later >
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: Permission denied
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
2 errors; 14 distinct packages available

I read thru https://github.com/gliderlabs/docker-alpine/issues/191, but we are not using authentication on proxy, so we cannot use any username/password anything. I tried with HTTP_PROXY_AUTH, but getting still the same.

msitruk commented 4 years ago

Got exaclty the same issue ! working for alpine:latest who use http and not alpine:edge who use https ...

stevenylai commented 4 years ago

Instead of running apk, can you try wget from your Dockerfile and see if there's any error? wget https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz

jpmorin commented 4 years ago

I have the same issue. I was going to create a new issues but stumble upon yours first. Here is the solution:

I learned from this issue that dl-cdn.alpinelinux.org does not support TLS at all

alpine:3.12 (latest as of august 24 2020)

$cat /etc/apk/repositories 
http://dl-cdn.alpinelinux.org/alpine/edge/main
http://dl-cdn.alpinelinux.org/alpine/edge/community

alpine:edge

$cat /etc/apk/repositories 
https://dl-cdn.alpinelinux.org/alpine/edge/main
https://dl-cdn.alpinelinux.org/alpine/edge/community

APK repositories url should be http only.

ronnicek commented 4 years ago

@jpmorin Kudos! Well done, but then bug is in alpine:edge docker image, since I am not changing repositories to https and they are there by default.

So really.. I put there RUN sed -i 's/https/http/' /etc/apk/repositories and it's working now.. but I feel it's a workaround more then the solution.

jpmorin commented 4 years ago

@ronnicek Yes, I did the same sed command in my Dockerfile to make it works.

FROM alpine:edge

RUN sed -i 's/https/http/' /etc/apk/repositories

RUN apk add curl

The thing is every previous version of alpine images had plain http repository urls. To be honest I don't know if it's a simple mistake that introduce the https urls or if it is a deliberate change to apk configuration...

garw commented 3 years ago

I do appreciate switching to https in general and I don't think that just replacing it again with http is a reasonable course of action. The root cause seems to be a bug in libfetch (that apk internally uses to fetch data from http): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220468

The bug ticket even has a patch attached that at least one commenter claims works. However, there seems to be no progress upstream. Maybe alpine can apply this on their libfetch version as an intermediate solution or push for it to get merged?

It also seems that this is now becoming more important as https is now the default in alpine:latest since Friday.

fabled commented 3 years ago

There has been multiple issues with libfetch and proxy + https. Hopefully those issues are fixed now in the latest apk-tools release which is pushed to edge. Could someone verify if apk 2.12.1 works?

seriouz commented 3 years ago

Editing all of our pipelines to replace https with http ist no real solution. Please upgrade to the new libfetch.

devonwarren commented 3 years ago

I also ran into this issue behind a corporate http proxy, not using proxy authentication but https traffic is decrypted/re-encrypted using an internal CA so I can understand why that wouldn't be verified. Here are the 2 solutions I found:

Copy/override the /etc/ssl/certs/ca-certificates.crt with your internal CA chain file
Run sed -i 's/https\:\/\//http\:\/\//g' /etc/apk/repositories (this may stop working if they enforce https only at mirror level)

It would be nice if the ca-certificates apk was installed by default so you could use that import the certs

dHannasch commented 3 years ago

* Copy/override the /etc/ssl/certs/ca-certificates.crt with your internal CA chain file

* Run `sed -i 's/https\:\/\//http\:\/\//g' /etc/apk/repositories` (this may stop working if they enforce https only at mirror level)

I'm using https://gitlab.com/shell-bootstrap-scripts/shell-bootstrap-scripts/-/blob/master/before_script.yaml#L243 to bootstrap by manually adding the certificate if the image doesn't have ca-certificates installed.

    - cat $PROXY_CA_PEM >> /etc/ssl/certs/ca-certificates.crt
    - if [ "$(tail -n $PROXY_CA_PEM_NUM_LINES /etc/ssl/certs/ca-certificates.crt)" != "$(cat $PROXY_CA_PEM)" ]; then false; fi

This allows apk add to work without messing with /etc/apk/repositories on alpine:3.13.1, but not on alpine:edge.

maurizio-lattuada commented 3 years ago

FYI, the issue referenced above related to libfetch seems it has been solved, at least when using Alpine 3.14.0

minecraftchest1 commented 3 years ago

I am still seeing this issue.

benispeti commented 3 years ago

I can still see the same issue on Alpine 3.14.0. I'm behind a corporate proxy which doesn't require any authentication. Adding the internal certificate to /etc/ssl/certs/ca-certificates.crt didn't help me.

My Dockerfile:

FROM alpine:3.14
RUN apk update

This is the log what I can see:

#5 [2/2] RUN apk update
#5 sha256:a1332c9af190e6718585316bf5b8f7a6f06f6207cc6c4d67c961fbdd95725e5d
#5 0.503 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
#5 5.509 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
#5 5.509 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/main: temporary error (try again later)
#5 5.509 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
#5 10.52 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/community: temporary error (try again later)
#5 10.52 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
#5 10.52 2 errors; 14 distinct packages available
#5 ERROR: executor failed running [/bin/sh -c apk update]: exit code: 2

FLAGLORD commented 3 years ago

I can still see the same issue on Alpine 3.14.0. I'm behind a corporate proxy which doesn't require any authentication. Adding the internal certificate to /etc/ssl/certs/ca-certificates.crt didn't help me.

My Dockerfile:

FROM alpine:3.14
RUN apk update

This is the log what I can see:

#5 [2/2] RUN apk update
#5 sha256:a1332c9af190e6718585316bf5b8f7a6f06f6207cc6c4d67c961fbdd95725e5d
#5 0.503 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
#5 5.509 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
#5 5.509 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/main: temporary error (try again later)
#5 5.509 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
#5 10.52 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/community: temporary error (try again later)
#5 10.52 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
#5 10.52 2 errors; 14 distinct packages available
#5 ERROR: executor failed running [/bin/sh -c apk update]: exit code: 2

Have you solved this problem?

rubenmanzano commented 3 years ago

I can still see the same issue on Alpine 3.14.0. I'm behind a corporate proxy which doesn't require any authentication. Adding the internal certificate to /etc/ssl/certs/ca-certificates.crt didn't help me. My Dockerfile:

FROM alpine:3.14
RUN apk update

This is the log what I can see:

#5 [2/2] RUN apk update
#5 sha256:a1332c9af190e6718585316bf5b8f7a6f06f6207cc6c4d67c961fbdd95725e5d
#5 0.503 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
#5 5.509 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
#5 5.509 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/main: temporary error (try again later)
#5 5.509 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
#5 10.52 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/community: temporary error (try again later)
#5 10.52 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
#5 10.52 2 errors; 14 distinct packages available
#5 ERROR: executor failed running [/bin/sh -c apk update]: exit code: 2

Have you solved this problem?

FROM alpine:3.14 RUN sed -i 's/https/http/' /etc/apk/repositories RUN apk update

Paraphraser commented 3 years ago

Still happening - see:

"sed" patch seems to work.

senarvi commented 3 years ago

For me, switching to http doesn't help. This is my Dockerfile:

FROM alpine:3.14
RUN sed -i 's/https\:\/\//http\:\/\//g' /etc/apk/repositories
RUN apk update

This is what happens:

Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM alpine:3.14
 ---> 3e8172af00ce
Step 2/3 : RUN sed -i 's/https\:\/\//http\:\/\//g' /etc/apk/repositories
 ---> Running in 5c4bad3836f3
Removing intermediate container 5c4bad3836f3
 ---> 70e5ba29199e
Step 3/3 : RUN apk update
 ---> Running in b7298b9f226e
fetch http://dl-cdn.alpinelinux.org/alpine/v3.14/main/armv7/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.14/community/armv7/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.14/main: temporary error (try again later)
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.14/community: temporary error (try again later)
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
2 errors; 14 distinct packages available
The command '/bin/sh -c apk update' returned a non-zero code: 2

I've tried it a few times over the past couple of weeks, so it doesn't seem like a temporary error. I can access those URLs from the command line from the same host.

senarvi commented 3 years ago

A quick update. I tried to debug this by running commands in the Dockerfile. It seems that the DNS resolver doesn't work at all. This Dockerfile:

FROM alpine:3.14
RUN nslookup www.google.com

results in this error:

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM alpine:3.14
 ---> 3e8172af00ce
Step 2/2 : RUN nslookup www.google.com
 ---> Running in a91b37b1e326
nslookup: clock_gettime(MONOTONIC) failed
The command '/bin/sh -c nslookup www.google.com' returned a non-zero code: 1

I found a couple of threads that might be related:

https://github.com/haugene/docker-transmission-openvpn/issues/1726 https://github.com/haugene/docker-transmission-openvpn/issues/1564

I tried those suggestions that I could, including setting the DNS server to 8.8.8.8 an disabling IPv6, but no success yet.

I should mention that I'm running this on a Raspberry Pi.

senarvi commented 3 years ago

A solution was pointed out to me in another thread. For me, it was enough to install libseccomp2 in the host OS:

sudo apt-key adv --keyserver hkps://keyserver.ubuntu.com:443 --recv-keys 04EE7237B7D453EC 648ACFD622F3D138
echo "deb http://httpredir.debian.org/debian buster-backports main contrib non-free" | sudo tee -a "/etc/apt/sources.list.d/debian-backports.list"
sudo apt update
sudo apt install libseccomp2 -t buster-backports

The issue is related to Docker and Raspberry Pi is detailed in this blog post by Sam Cater.

rvillane commented 3 years ago

I just tried using alpine:3.14 and without the sed -i 's/https/http/' /etc/apk/repositories trick and got the same error:

=> ERROR [6/6] RUN apk -U upgrade && apk add --no-cache --virtual .gyp         python3         make         g++     && npm install -g nodemon     &&   1.2s
------                                                                                                                                                       
 > [6/6] RUN apk -U upgrade && apk add --no-cache --virtual .gyp         python3         make         g++     && npm install -g nodemon     && npm install --production     && npm cache clean --force     && apk del .gyp     && rm -rf /usr/local/lib/node_modules/npm:                                                 
#10 0.301 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz                                                                      
#10 0.666 140413134482248:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914:                
#10 0.669 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/main: Permission denied                                                                         
#10 0.669 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
#10 0.669 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
#10 0.835 140413134482248:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914:
#10 0.838 ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/community: Permission denied
#10 0.838 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
#10 0.840 OK: 7 MiB in 16 packages
#10 0.845 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
#10 1.013 140585732721480:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914:
#10 1.016 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: Permission denied

running the sed command above made it work fine with HTTP.

pbertin-oneex commented 3 years ago

Same issue, trying HTTP/HTTPS on all the public docker images available on Dockerhub.

I am using docker on AlpineLinux 3.13.4, and running apk update fetches just fine. Is the minirootfs missing / having outdated packages essential to fetch?

@rvillane which docker version? Just try with your setup and not working here (both HTTP/HTTPS). Mine is 20.10.3.

rvillane commented 3 years ago

@pbertin-oneex I'm using docker 20.10.7 in both client and server.

had you tried adding the following line in your Dockerfile before making any calls to APK ?

RUN sed -i 's/https/http/' /etc/apk/repositories

Coderdude112 commented 2 years ago

@rvillane I tried what you said to add and still got errors within the first three lines of my dockerfile

FROM alpine:3.14

RUN sed -i 's/https/http/' /etc/apk/repositories

RUN apk update \
  && apk add --no-cache \
    gcc \
    g++ \
    libffi-dev \
    musl-dev \
    postgresql-dev \
    yarn \
  && python -m pip install --upgrade pip

benispeti commented 2 years ago

Finally this thing looks to be working on my end.

FROM alpine:3.15
ARG PROXY="<your proxy URL comes here>"
RUN sed -i 's/https/http/' /etc/apk/repositories
RUN export HTTP_PROXY=$PROXY && export HTTPS_PROXY=$PROXY && \
    apk update

pbertin-oneex commented 2 years ago

On my end too, without any changes to my setups (same dockerfile, same docker version, same host config, tested on arch / ubuntu / alpine), fetch works today in http and https (without the use of proxy in my case). Http server issue / configuration then?

Coderdude112 commented 2 years ago

What did you use for the proxy server @benispeti?

I'm still getting the error

benispeti commented 2 years ago

What did you use for the proxy server @benispeti?

I'm still getting the error

I had this issue behind my corporate proxy, so I used it's URL here. You should use your company's proxy URL.

Coderdude112 commented 2 years ago

Gotcha, yeah I'm trying to do this at home and still having problems. Is this thread exclusive to having this problem behind a corporate proxy?

Gandulf78 commented 2 years ago

Gotcha, yeah I'm trying to do this at home and still having problems. Is this thread exclusive to having this problem behind a corporate proxy?

same issue here on my home network

Paraphraser commented 2 years ago

Are you running Buster? If yes, have you upgraded libseccomp2? If no, see instructions.

There's background info here.

Also, Mosquitto seemed to need the sed patch for https to http. See [Dockerfile line 5](https://github.com/SensorsIot/IOTstack/blob/master/.templates/mosquitto/Dockerfile]. That's mentioned above here too so I'd try both.

Gandulf78 commented 2 years ago

Are you running Buster? If yes, have you upgraded libseccomp2? If no, see instructions.

Finally I succeeded to update my docker version. It is ok now.

skaldesh commented 2 years ago

I have this error now, needed to replace https with http

FROM alpine:3.15 AS final

# Install dependencies.
RUN sed -i 's/https/http/' /etc/apk/repositories
RUN apk add --no-cache \
    ca-certificates \
    docker

No proxy being used, but we build using docker buildx

kunlong-luo commented 2 years ago

Add some code to the dockerfile to solve this problem

RUN sed -i ‘s/https/http/’ /etc/apk/repositories
RUN apk add curl

OJFord commented 2 years ago

I have the same problem on Arch with libseccomp v2.5.4, all alpine versions that I tested (3.10 through 3.16), with and without TLS, with buildkit and not. With both docker v20.10.17 and podman 4.1.1.

Not on a corporate network, no proxy, I'm at home. It works in GH Actions though.

molitona commented 2 years ago

same as you arch too.. @OJFord

LeonardoHQ commented 2 years ago

Hello! I faced the same problem on alpine:3.16 with Fedora 36 as host. I found the solution here https://shaik-tech.blogspot.com/2018/07/docker-error-warning-ignoring-httpdl.html

I just created the daemon.json file (in the fedora host) with the following content

{
    "dns": ["8.8.8.8"]
}

in /etc/docker/daemon.json then restarted the docker service (just in case) with sudo systemctl restart docker.service and all good.

My Dockerfile looks like:

FROM alpine:3.16
RUN apk upgrade
RUN apk add python3 py3-pip neovim
etc
etc

Note that I do not use the sed trick.

drrobotic commented 2 years ago

to everyone who has still this problem and all suggested solutions dont work, please check your firewall rules. opening ports for dns(53) and https(443) alone via OUTPUT chain seems not enough, i also had to put the accept rules into the DOCKER-USER chain, after that https requests inside container worked again. its better to test container connectivity with ping/wget, i think the error message "temporary error (try again later)" from apk is very confusing. after allowing dns only there comes a much better hint "network error (check Internet connection and firewall)".

kyberorg commented 2 years ago

@drrobotic agreed. In my case it was rule that redirected all incoming traffic on port 443 to another port. Problem was solved by specifying WAN interface only.

SamuelLHuber commented 1 year ago

I was able to fix the issue in a microk8s Kubernetes Cluster by specifying dnsPolicy and hostNetwork in the CronJob spec.

While normal pods created with kubectl run had no issues to run apk update the pod created by the cronjob did. That fixed it for me. Haven't validated if dnsPolicy alone would be enough, but maybe you don't need the hostNetwork: true

To learn what is actually happening check the Kubernetes Docs for dns-pod-service

spec:
  schedule: {{ .Values.backup.schedule | quote }}
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: "OnFailure"
          hostNetwork: true
          dnsPolicy: ClusterFirstWithHostNet

r2evans commented 9 months ago

I'm getting the permission-denied error. My firewall is fine, my docker setup is unchanged over many years. This is reproducible across different networks, though I'm finding it fails more when the CDN resolves to the 146.* addresses for dl-cdn.alpinelinux.org but not for the 151.* resolution. The below are within the 146.* area.

This inconsistency to me suggests it is not necessarily with the internal libraries or ca-certificates (I tested after adding that package, too). Could it be something with the CDN?

All in-alpine commands below are based on this container and cmdline:

root@franz:~# docker images | grep -E 'REPO|alpine'
REPOSITORY                                                          TAG                            IMAGE ID       CREATED         SIZE
alpine                                                              latest                         05455a08881e   3 weeks ago     7.38MB
root@myhost:~# docker run -it --rm alpine sh

/ # cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.19.1
PRETTY_NAME="Alpine Linux v3.19"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

Inconsistency

The behavior is inconsistent and changes on each invocation (even within the same container).

/ # apk add --no-cache  ca-certificates
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.19/main: Permission denied
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.19/community: Permission denied
ERROR: unable to select packages:
  ca-certificates (no such package):
    required by: world[ca-certificates]

/ # apk add --no-cache  ca-certificates
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/1) Installing ca-certificates (20230506-r0)
Executing busybox-1.36.1-r15.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 8 MiB in 16 packages

No delay between attempts (there were a half-dozen attempts before the first shown).

Checking certs

/ # sed -i s/https/http/ /etc/apk/repositories
/ # apk add openssl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/1) Installing openssl (3.1.4-r5)
Executing busybox-1.36.1-r15.trigger
OK: 8 MiB in 16 packages

With that temporary hack (I don't yet accept going http:// as a secure and acceptable final solution):

/ # openssl version                                                                                      
OpenSSL 3.1.4 24 Oct 2023 (Library: OpenSSL 3.1.4 24 Oct 2023)  

/ # echo | openssl s_client -connect dl-cdn.alpinelinux.org:443 -showcerts
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 324 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

And in the same container, if I wait a moment and try again,

/ # echo | openssl s_client -connect dl-cdn.alpinelinux.org:443 -showcerts
CONNECTED(00000003)                                                                                      
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1                                                                                          
depth=1 C = US, O = Let's Encrypt, CN = R3                                                               
verify return:1                                                                                          
depth=0 CN = dl-cdn.alpinelinux.org
verify return:1
---
Certificate chain
 0 s:CN = dl-cdn.alpinelinux.org
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 29 16:57:18 2024 GMT; NotAfter: Apr 28 16:57:17 2024 GMT
-----BEGIN CERTIFICATE-----
MIIE+TCCA+GgAwIBAgISA1hz6FwMJ3EYQna99ui1aKbaMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
...

(A completely-valid SSL connection.)

Changing DNS

Changing to quad-8 for DNS resolution does not fix it.

/ # sed  -E 's/nameserver.*/nameserver 8.8.8.8/g' /etc/resolv.conf > /tmp/resolv.conf
/ # cat /tmp/resolv.conf > /etc/resolv.conf 
/ # cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 8.8.8.8

/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.19/main: Permission denied
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.19/community: Permission denied
4 unavailable, 0 stale; 15 distinct packages available

wget failure

wget reports that the connection is reset by the peer?

/ # wget -S https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
Connecting to dl-cdn.alpinelinux.org (146.75.30.132:443)
ssl_client: SSL_connect
wget: error getting response: Connection reset by peer

lhartmann commented 8 months ago

Got similar errors, along with many SSL failues on curl and wget, using alpine 3.18 and 3.19.

POTENTIAL CAUSE: Missing ca-certificates and symlink /usr/lib/ssl/certs -> /etc/ssl/certs.

The Error

~ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.19/main: Permission denied
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:80000002:system library:file_open:No such file or directory:providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs)
489BE4F0E87F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:
WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.19/community: Permission denied
4 unavailable, 0 stale; 73 distinct packages available

The Fix

Change /etc/apk/repositories from HTTPS to HTTP
Install the ca-certificates package
Run update-ca-certificates
Change /etc/apk/repositories back from HTTP to HTTPS
Create /usr/lib/ssl/
Create symlink /usr/lib/ssl/certs -> /etc/ssl/certs

~ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
v3.19.1-169-g5ed82648e0c [https://dl-cdn.alpinelinux.org/alpine/v3.19/main]
v3.19.1-180-g5674b95fd09 [https://dl-cdn.alpinelinux.org/alpine/v3.19/community]
OK: 23060 distinct packages available
~ #

lordloc commented 3 months ago

Up until today August 2024, I still have this problem. This is the context:

Our application is deployed on Kubernetes Pod
This issue rarely happens, like 1-2 fail for 100 success.

This is the error logs

│ yatai-66b05443037de60b6d934bdb--startup-mon--1-nv7nk + apk add --no-cache curl                                                                                                                                   
│ yatai-66b05443037de60b6d934bdb--startup-mon--1-nv7nk fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/main/x86_64/APKINDEX.tar.gz                                                                               
│ yatai-66b05443037de60b6d934bdb--startup-mon--1-nv7nk fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/community/x86_64/APKINDEX.tar.gz                                                                          
│ yatai-66b05443037de60b6d934bdb--startup-mon--1-nv7nk WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.20/main: Permission denied

Base image is a mirror image of docker.io/library/bash:latest, I guess we are using up-to-date version of that image as well.

Because this happens intermittently, I think it's mostly related to tls certificate verification, but the real root cause is still unknown, because the complexity of the network layers in the company.

danny-huang-openfind commented 1 month ago

I’m using Alpine 3.17 (node:18.16.0-alpine3.17), but I still occasionally encounter this problem, though it doesn't happen every time.

#13 [builder 2/7] RUN apk add --no #-cache --verbose     bash     git     && corepack enable     && yarn global add zx -D
#13 0.230 fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
#13 15.69 fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
#13 15.69 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/main: Permission denied
#13 17.10 ERROR: unable to select packages:
#13 17.15   bash (no such package):
#13 17.15     required by: world[bash]
#13 17.15   git (no such package):
#13 17.15     required by: world[git]
#13 ERROR: process "/bin/sh -c apk add --no-cache --verbose     bash     git     && corepack enable     && yarn global add zx -D" did not complete successfully: exit code: 2

alpinelinux / docker-alpine