search

gliderlabs / docker-alpine

Alpine Linux Docker image. Win at minimalism!
http://gliderlabs.viewdocs.io/docker-alpine
BSD 2-Clause "Simplified" License
5.71k stars 531 forks source link

Repository problem? #279

Closed Skorpyon closed 7 years ago

Skorpyon commented 7 years ago

http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz

Error 503 Connection timed out Connection timed out Guru Mediation: Details: cache-hhn1533-HHN 1494522331 3223824506 Varnish cache server

christhomas commented 6 years ago

I am still getting this error, I don't think I should manually configure this, it should be resolved on the server side with no changes my side

shaider0802 commented 6 years ago

I was having this issue on Windows 10, resolved it by following peternormann 's suggestion to point the DNS settings of DockerNAT to google's..

wisprp commented 6 years ago 
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

Worked fine 15 min ago, but now I'm getting this error.

KristopherWindsor commented 6 years ago

I am getting this error now. http://dl-cdn.alpinelinux.org/alpine/v3.6/main says:

Error 503 Backend is unhealthy
Backend is unhealthy

Guru Mediation:
Details: cache-sjc3146-SJC 1518737504 1886963746

Varnish cache server
wisprp commented 6 years ago

Back online.

Francesco149 commented 6 years ago

changing /etc/resolv.conf to point to google DNSes fixed the issue for me (gentoo linux, docker 17.09.1-ce)

dwaiba commented 6 years ago

3.7 mirrors are down again

barthwalsamarth commented 6 years ago

Getting this error / # apk add vim WARNING: Ignoring APKINDEX.70c88391.tar.gz: No such file or directory WARNING: Ignoring APKINDEX.5022a8a2.tar.gz: No such file or directory ERROR: unsatisfiable constraints: vim (missing): required by: world[vim] How can I fix it?

ok found the solution

echo http://dl-cdn.alpinelinux.org/alpine/edge/testing >> /etc/apk/repositories apk update apk add vim this worked

roffe commented 6 years ago

Been seeing this a couple of times over the past days making my automatic builds break :(

fr33ek commented 6 years ago

Same now.

ChrisChiasson commented 6 years ago

Getting same error as everyone else in docker. I'm on google compute engine in a VM on Ubuntu 16.04 LTS. In Chrome from my workstation: http://dl-cdn.alpinelinux.org/alpine/v3.7/main Error 503 Backend is unhealthy Backend is unhealthy

Guru Mediation: Details: cache-mia17625-MIA 1521467300 1442441019

Varnish cache server

ghost commented 6 years ago

Same here, now

dewey commented 6 years ago

The mirror of the latest mirrors is up in case anyone wants to switch over:

http://mirror.yandex.ru/mirrors/alpine/MIRRORS.txt

ChrisChiasson commented 6 years ago

As requested:

<user>@<machine>:~/mingw-w64-clang$ curl -vvvv http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.0.249...
* Connected to dl-cdn.alpinelinux.org (151.101.0.249) port 80 (#0)
> GET /alpine/v3.7/main/x86_64/APKINDEX.tar.gz HTTP/1.1
> Host: dl-cdn.alpinelinux.org
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 503 Backend is unhealthy
< Server: Varnish
< Retry-After: 0
< Content-Type: text/html; charset=utf-8
< Content-Length: 455
< Accept-Ranges: bytes
< Date: Mon, 19 Mar 2018 13:51:15 GMT
< Via: 1.1 varnish
< Connection: close
< X-Served-By: cache-mdw17322-MDW
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1521467475.266664,VS0,VE0
< 
{ [455 bytes data]
100   455  100   455    0     0   8315      0 --:--:-- --:--:-- --:--:--  8425
dewey commented 6 years ago

It's back up for me now!

ChrisChiasson commented 6 years ago

The main started working again.

lorintotahh commented 6 years ago

same here :(

wstrange commented 6 years ago

http://dl-cdn.alpinelinux.org/alpine/v3.7/main is down again

JPWKU commented 6 years ago

I would like to report that we are also seeing timeouts updating against dl-cdn.alpinelinux.org. I have sidestepped the issue by adding references to one of the mirrors in our Dockerfile.

JamesKyburz commented 6 years ago

Mirror for 3.6 seems to be done at the moment :(

http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

zfeng8 commented 6 years ago

Any update on this? I have used a mac computer to do the work, it works fine. However, every time I switch to an Ubuntu system, the problem occurs.

Necmttn commented 6 years ago

image looks like down again ?

JoaoCarabetta commented 6 years ago

I am still getting this error in Ubuntu. As @zfeng8 mentioned, it works fine on mac but fails on an Ubuntu. I also tried a bunch of mirrors, slashes and restarting the docker daemon.

Does anyone know how to fix it?

zfeng8 commented 6 years ago

@JoaoCarabetta if you have a mac computer, try to build the image and somehow move to the ubuntu. Then you should be fine

johnkoehn commented 6 years ago

Getting this error currently :(

LuckyTyagi commented 6 years ago

I'm facing 'timed out' error on this. Following Kubernetes up and running book.

# make build
container: kuard-build
Sending build context to Docker daemon  11.23MB
Step 1/10 : FROM golang:1.7-alpine
 ---> 974aa102bae2
Step 2/10 : WORKDIR /data
 ---> Using cache
 ---> 03e6eee40878
Step 3/10 : ENV GOPATH /data/go
 ---> Using cache
 ---> a4263084a348
Step 4/10 : ENV npm_config_cache /data/npm_cache
 ---> Using cache
 ---> 1d1067eb3a33
Step 5/10 : ARG ALL_ARCH
 ---> Using cache
 ---> c5ed0dff6d95
Step 6/10 : ENV ALL_ARCH ${ALL_ARCH}
 ---> Using cache
 ---> 50872afbf603
Step 7/10 : RUN for ARCH in ${ALL_ARCH}; do       ln -s -f "/data/std/${ARCH}" "/usr/local/go/pkg/linux_${ARCH}_static" ;     done
 ---> Using cache
 ---> 98b54c180d31
Step 8/10 : RUN apk update && apk upgrade && apk add --no-cache git nodejs bash
 ---> Running in 549a74a0a473
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: operation timed out
WARNING: Ignoring APKINDEX.167438ca.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/community: operation timed out
WARNING: Ignoring APKINDEX.a2e6dac0.tar.gz: No such file or directory
2 errors; 12 distinct packages available
The command '/bin/sh -c apk update && apk upgrade && apk add --no-cache git nodejs bash' returned a non-zero code: 2
make: *** [.kuard-build-container] Error 2
[root@rsvm63 kuard]#
ncopa commented 6 years ago

@LuckyTyagi I can not reproduce it. Can you please tell me what you get from:

curl --head  http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
LuckyTyagi commented 6 years ago

@ncopa It goes like this:

# curl --head  http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
HTTP/1.1 200 OK
Server: nginx
Content-Type: application/octet-stream
Last-Modified: Mon, 30 Apr 2018 18:50:38 GMT
ETag: "5ae7657e-a5b18"
Via: 1.1 varnish
Accept-Ranges: bytes
Date: Mon, 07 May 2018 13:01:59 GMT
Via: 1.1 varnish, 1.1 mc-loc-web-gw.alcatel-lucent.com
Age: 13700
X-Served-By: cache-jfk8145-JFK, cache-atl6228-ATL
X-Cache: HIT, HIT
X-Cache-Hits: 1, 1
X-Timer: S1525698119.381417,VS0,VE22
Content-Length: 678680
neclimdul commented 6 years ago

I was just having this problem repeatedly over the last couple hours. I fired up wireshark to see if there was something to be seen and it started working. schroedinbug

Because I didn't confirm it I don't know that its the same problem but I've seen a similar problem with fastly mirrors quite often before though. Sometimes its alpine, sometimes its npmjs.org. Wireshark shows the TCP session stuck with fastly sending packets sized very closed to 1500 MTU with a no fragment flag and the linux bridge failing to route because of a lower MTU. Repeat until it eventually dies.

Its especially easy to recreate in k8s because k8s sets a lower MTU on the bridge to to allow for routing overlay network traffic and I can consistently create the problem using DinD to build containers in k8s.

The log looks something like this:

... misc traffic 
ack:                   container -> node -> fastly
Transmission:          container  X node <- fastly (2924 no fragment)
ICMP unreachable:                   node -> fastly
Transmission:          container  X node <- fastly (1496 no framgment)
ICMP unreachable:                   node -> fastly
Transmission:          container  X node <- fastly (1496 no framgment)
.... repeat

I have a saved wireshark packet logs of this happening on a k8s node though its talking to npm's fastly mirror because I was using the workaround to bypass the alpine fastly mirrors. Its the same behavior though.

LuckyTyagi commented 6 years ago

Strange that nobody was able to reproduce the issue or provide the solution even after these many days..!!

neclimdul commented 6 years ago

there are half a dozen similar issues so i think people are reproducing it just not the people that can fix it.

ncopa commented 6 years ago

@neclimdul thank you for sharing that wireshark info which explains what is going on. It looks like it is PMTU that is broken. My guess is that the icmp packet with DF set if filtered by something along the way. It could be your ISP or it could be fastly. In either case, this is not something can can be fixed by changing anything in the alpine linux docker image.

ncopa commented 6 years ago

I was able to reproduce it. I ran alpine in a docker container on my linux host. I manually set mtu on the veth interface to 1450, and apk update just hangs. Switching repository to dl-4.alpinelinux.org makes it work, and it also sends ICMP needs frag. So it looks like fastly is filtering out ICMP need frag packets and break PMTU by doing so.

Again, this is not something Alpine specific and I am pretty sure this affects anything using fastly and should be reported to them.

neclimdul commented 6 years ago

Great, that matches what I've been seeing and yes it does affect other things as the log I've got stored is actually the npm js mirror on fastly. Someone connected to alpine's infrastructure might have better contacts to work with Fastly but lacking a better method I shot them a tweet for now.

https://twitter.com/neclimdul/status/994274338072248320

ncopa commented 6 years ago

I sent them a support mail, but they claim nothing is wrong with their network. Then they recommended the users to set the MTU manually. A better workaround would be to configure tcp mss clamping.

neclimdul commented 6 years ago

The MTU is manually set...

neclimdul commented 6 years ago

FWIW, opened the wireshark documented earlier. Checked the SYN request for the session, MSS set to 1460. so... not sure watch options there are.

JoFAM commented 6 years ago

I'm having the issue again with http://nl.alpinelinux.org and it suddenly started even though my dockers built perfectly fine this morning. The warnings about temporary errors point to the main repo dl-cdn.alpinelinux.org . Changing the repo to any other mirror gives the exact same errors.

The problem was solved after restarting Docker, but the problem is specific to alpine. Debian-based containers didn't have any problem.

Here's (part of) the docker output:

Step 4/6 : RUN echo '@edge http://nl.alpinelinux.org/alpine/edge/community' >> /etc/apk/repositories
      && echo '@edge http://nl.alpinelinux.org/alpine/edge/main' >> etc/apk/repositories
      && apk add --upgrade --no-cache apk-tools@edge
 ---> Running in 0b9d70ecba5a
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://nl.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://nl.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://nl.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: Ignoring http://nl.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: The repository tag for world dependency 'apk-tools@edge' does not exist
ERROR: Not committing changes due to missing repository tags. Use --force-broken-world to override.
zment commented 6 years ago

Down again :( With mirrors, alpine is pullable, but cannot apk add openldap

fetch http://dl-2.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz ERROR: unsatisfiable constraints: openldap (missing): required by: world[openldap]

reeshabh90 commented 6 years ago

I am facing the issue with alpine v3.4. Any solutions?

image

reeshabh90 commented 6 years ago

My Solutions: Configure Docker DNS on UBUNTU, it will fix the problem.

Refer the link below for reference:

https://development.robinwinslow.uk/2016/06/23/fix-docker-networking-dns/

publicocean0 commented 6 years ago

same problem

publicocean0 commented 6 years ago

in my case no solution works (reboot or dns config in daemon.json)

neclimdul commented 6 years ago

I'm guessing the DNS fixes some people are getting shake up the IP for the fastly mirror being used and maybe get a different path avoiding routers causing the ICMP issue. Using a global DNS versus a local one can really mess up CDNs localization so that's a very real possibility.

@publicocean0 a couple people have reached out to fastly now about this and they don't seem to be acknowledging the problem. Maybe reach out as well? I wish migrating off alpine was even an option but since I see the same problem with node and other fastly hosted projects I'm just :confounded:

LugaMuga commented 6 years ago

I found that docker alpine in Docker cannot resolve repo url when I've added iptables rule:

sudo iptables -t nat -F sudo iptables -t mangle -F iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

And fix this issue can only remove all rules from iptables and restat docker:

sudo iptables -F && sudo iptables -X && sudo iptables -S sudo service docker restart

knaggit commented 6 years ago

For everyone who's still facing that problem. It seems, there are DNS probs in Alpine DNS records. It affects somehow other repositories as well. I tried other repos and still had the problem. On which URL everything seems fine is, when you RUN sed before getting packages as below:

RUN sed -i -e 's/dl-cdn/dl-4/' /etc/apk/repositories && \
    apk add --no-cache \
        bash \
        build-base \

I read all the threads and finally that got me help. Good luck!

igwejk commented 6 years ago

@andyshinn This issue has come up again, is there anything you can do about it?

fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/main: IO ERROR
WARNING: Ignoring APKINDEX.adfa7ceb.tar.gz: No such file or directory
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/community: IO ERROR
WARNING: Ignoring APKINDEX.efaa1f73.tar.gz: No such file or directory
2 errors; 13 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/main: IO ERROR
WARNING: Ignoring APKINDEX.adfa7ceb.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/community: IO ERROR
WARNING: Ignoring APKINDEX.efaa1f73.tar.gz: No such file or directory

image

igwejk commented 6 years ago

image

Can you recommend a way to ensure this issue doesn't occur frequently?

us108y commented 6 years ago

I am also getting the same Error as of today (Aug -09-2018)

FROM alpine

fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

`curl -v -s -o /dev/null http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz

while other cdn works

T0T4R4 commented 6 years ago

Same issue on ubuntu server 18.04 with Docker version 18.06.0-ce, build 0ffa825

Surprisingly, It worked on my fedora 27 workstation with Docker version 1.13.1, build 30c1ca9-unsupported

The error message is :

---> cc8b89f74863
Step 5/34 : RUN     apk add --update --no-cache nginx nodejs nodejs-npm git curl wget gcc ca-certificates                                     python-dev py-pip musl-dev libffi-dev cairo supervisor bash                                     py-pyldap py-rrd                                                                 
&&        wget -q -O /etc/apk/keys/sgerrand.rsa.pub   https://raw.githubusercontent.com/sgerrand/alpine-pkg-glibc/master/sgerrand.rsa.pub              
&&        wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.26-r0/glibc-2.26-r0.apk                &&        apk add --no-cache  glibc-2.26-r0.apk                                                                        
&&        rm glibc-2.26-r0.apk                                                                                         
&&        adduser -D -u 1000 -g 'www' www                                                                              
&&        pip install -U pip pytz gunicorn six --no-cache-dir                                                          
&&  npm config set proxy http://proxy.usq.edu.au:8080                                
&&  npm config set https-proxy http://proxy.usq.edu.au:8080                              
&&        npm install -g wizzy                                                                                         
&&        npm cache clean --force

 ---> Running in fb046f39414f
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
ERROR: unsatisfiable constraints:
  bash (missing):
    required by: world[bash]
  ca-certificates (missing):
    required by: world[ca-certificates]
  cairo (missing):
    required by: world[cairo]
  curl (missing):
    required by: world[curl]