search

actions / runner-images

GitHub Actions runner images
MIT License
9.17k stars 2.84k forks source link

azure.archive.ubuntu.com is unstable again #7048

Closed RadxaYuntian closed 1 year ago

RadxaYuntian commented 1 year ago

Description

Similar to recent reported issue #6894 #6909 #6913, runner is not able to connect to azure.archive.ubuntu.com to fetch all packages. apt update appears unaffected.

Platforms affected

Runner images affected

Image version and build link

ubuntu-22.04: https://github.com/radxa-build/rock-4se/actions/runs/4102124998

Is it regression?

No

Expected behavior

apt install completes successfully.

Actual behavior

apt install could not complete due to network related issue.

Repro steps

Rerun failed job.

jlaine commented 1 year ago

I can confirm I'm experiencing the same issue.

provokateurin commented 1 year ago

Same here

EmilioCarasa commented 1 year ago

Im experiencing this same issue with Ubuntu 18.04

KristapsT commented 1 year ago

Experiencing the same on Ubuntu 22.04 on Large Runners.

probeonstimpack commented 1 year ago

Same here

alkino commented 1 year ago

Windows Server 2022 affected too. Ubuntu 20.04 too

jlaine commented 1 year ago

Hi @ddobranic ,

Hello, it is something out of our control. As you can see previous time it was an issue with Canonical servers.

Given how frequent this issue is, doesn't it highlight the limits of the "caching proxy" approach? Wouldn't it be more reliable to set up an actual mirror so that upstream downtime only affects syncing the mirror, not CI runs?

akbast commented 1 year ago

Same here

bdo commented 1 year ago

This affects us pretty badly here at Warner Bros. Discovery!

florian-hoenicke commented 1 year ago

We also have issues. What is a workaround we could do?

karmab commented 1 year ago

someone mentioned sed -i 's/azure\.//' /etc/apt/sources.list before apt-get update worth a shot if you cant wait for azure network issues to be fixed

nstng commented 1 year ago

Same issue here using the ubuntu sources, e.g.,

Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to archive.ubuntu.com:80

edit: ok, the link above https://github.com/actions/runner-images/issues/675#issuecomment-1382027536 already explains the connection

mikhailkoliada commented 1 year ago

Hello! It has been handed over to the appropriate team! We will post the updates.

bdo commented 1 year ago

sed -i 's/azure.//' /etc/apt/sources.list

Thanks @karmab, but this didn't work for me.

RooGhz720 commented 1 year ago

same here can't fetch package

LiliDeng commented 1 year ago

We have created ubuntu support case, they will response within 15 mins.

AlexanderLanin commented 1 year ago

FYI setting apt to retry 10 times as suggested somewhere else does not help. It just takes longer to fail on a random package.

echo 'Acquire::Retries "10";' | sudo tee -a /etc/apt/apt.conf.d/80-retries > /dev/null

LiliDeng commented 1 year ago

canonical's reply This appears to be a downstream issue related to a current Ubuntu infra incident. Working on getting more information on root cause and ETTR. I'll update you shortly.

aqeelat commented 1 year ago

@LiliDeng could you please tell them to update https://status.canonical.com/

aqeelat commented 1 year ago

@mikhailkoliada Would we still get billed for the failing actions? I have multiple runs that took about 45 minutes on the apt step alone.

LiliDeng commented 1 year ago

@LiliDeng could you please tell them to update https://status.canonical.com/

I have asked this in the canonical case.

florian-hoenicke commented 1 year ago

@RadxaYuntian the 20.04 runner is also impacted

mefistotelis commented 1 year ago

When I click the failing link, I can download the deb no problem. Is the issue specific to connection between github and canonical?

LiliDeng commented 1 year ago

canonical's reply ` The request backlog has been terminated, and things should be returning to normal for the most part. We're still monitoring / working on determining the root cause.

Re: the status site, it looks like due to the flakey nature of the issue (as our apache connection limit was intermittently being exhausted), the status never toggled to down. I'll have IS look into adding the cloud archives and more rigid status alerting as a followup. `

LiliDeng commented 1 year ago

When I click the failing link, I can download the deb no problem. Is the issue specific to connection between github and canonical?

no, it is not related with github, when we try to install the package or apt update on ubuntu azure vm, it also has this issue.

aqeelat commented 1 year ago

It worked for a minute but it is broken again.

4c0n commented 1 year ago

Well it's definitely an issue between Github/Azure and Ubuntu. Other mirrors are working fine. It's a shame that there are no other mirrors configured by default, that way it wouldn't have such a big impact on everyone's CI process.

florian-hoenicke commented 1 year ago

Can you quickly post a workaround we can do until it is fixed? Something like the modification of the /etc/apt/sources.list to use another mirror?

rafrafek commented 1 year ago

Add this step before your apt-get install step. It's slow but it works:

      - name: Update apt - with apt-spy2 workaround
         run: |
           sudo gem install apt-spy2
           sudo apt-spy2 check
           sudo apt-spy2 fix --commit
           sudo apt-get update
aloisklink commented 1 year ago

Can you quickly post a workaround we can do until it is fixed? Something like the modification of the /etc/apt/sources.list to use another mirror?

The following works pretty well for me:

    - name: Set apt mirror
      # GitHub Actions apt proxy is super unstable
      # see https://github.com/actions/runner-images/issues/7048
      run: |
        # make sure there is a `\t` between URL and `priority:*` attributes
        printf 'http://azure.archive.ubuntu.com/ubuntu  priority:1\n' | sudo tee /etc/apt/mirrors.txt
        curl http://mirrors.ubuntu.com/mirrors.txt | sudo tee --append /etc/apt/mirrors.txt
        sudo sed -i 's/http:\/\/azure.archive.ubuntu.com\/ubuntu\//mirror+file:\/etc\/apt\/mirrors.txt/' /etc/apt/sources.list

It downloads the official mirrors.txt file from http://mirrors.ubuntu.com/mirrors.txt but we append http://azure.archive.ubuntu.com/ubuntu priority:1\n (note that there is a tab, not a space between the URL and priority:1) so that apt always tries the faster http://azure.archive.ubuntu.com/ubuntu mirror first, and only falls back to the other entries in mirrors.txt on failure.

See apt-transport-mirror(1) man-page for more details on how apt mirrors works.

RadxaYuntian commented 1 year ago

Issue appears to be resolved: https://github.com/radxa-build/rock-4se/actions/runs/4109805293

brunobertechini commented 1 year ago

This keeps coming back every once in a while :(

therealdwright commented 1 year ago

This is impacting us also - it's quite frustrating.

LiliDeng commented 1 year ago

This is impacting us also - it's quite frustrating.

Ubuntu case created https://canonical.force.com/ua/s/case/5008e00000CiHzTAAV/could-not-connect-to-azurearchiveubuntucom80-52147219192-connection-timed-out-ip-52147219192-80

do you see the repo server ip, is it also 52.147.219.192?

Flexba commented 1 year ago

Also impacting me - the IPs encountered are

LiliDeng commented 1 year ago

Also impacting me - the IPs encountered are

  • 51.107.8.243
  • 51.116.128.6 I have updated them into above case, thank you!
LiliDeng commented 1 year ago

@Flexba @therealdwright hi, has the issue resolved in your side?

Flexba commented 1 year ago

Yes, resolved for me

jaboutboul commented 1 year ago

I can confirm this is now resolved on Canonical’s end

qaiserali commented 1 year ago

This is also impacting us. Any fix for it?

LiliDeng commented 1 year ago

This is also impacting us. Any fix for it?

is still on-going? please copy the error message

qaiserali commented 1 year ago

This is also impacting us. Any fix for it?

is still on-going? please copy the error message

We are getting the following error

`Unable to connect to azure.archive.ubuntu.com:http: Err:19 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 xxd amd64 2:8.1.2269-1ubuntu5.13 Unable to connect to azure.archive.ubuntu.com:http: Err:20 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 vim-common all 2:8.1.2269-1ubuntu5.13 Unable to connect to azure.archive.ubuntu.com:http: Err:26 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-cloud-tools-common all 5.4.0-147.164 Unable to connect to azure.archive.ubuntu.com:http: Err:27 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-tools-common all 5.4.0-147.164 Unable to connect to azure.archive.ubuntu.com:http: Fetched 9248 kB in 30s (304 kB/s)

[stderr] buntu3.21_amd64.deb Unable to connect to azure.archive.ubuntu.com:http: E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/libudev1_245.4-4ubuntu3.21_amd64.deb Unable to connect to azure.archive.ubuntu.com:http: E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd-sysv_245.4-4ubuntu3.21_amd64.deb Unable to connect to azure.archive.ubuntu.com:http:

E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/l/linux/linux-tools-common_5.4.0-147.164_all.deb Unable to connect to azure.archive.ubuntu.com:http: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?`

LiliDeng commented 1 year ago

This is also impacting us. Any fix for it?

is still on-going? please copy the error message

We are getting the following error

`Unable to connect to azure.archive.ubuntu.com:http: Err:19 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 xxd amd64 2:8.1.2269-1ubuntu5.13 Unable to connect to azure.archive.ubuntu.com:http: Err:20 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 vim-common all 2:8.1.2269-1ubuntu5.13 Unable to connect to azure.archive.ubuntu.com:http: Err:26 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-cloud-tools-common all 5.4.0-147.164 Unable to connect to azure.archive.ubuntu.com:http: Err:27 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-tools-common all 5.4.0-147.164 Unable to connect to azure.archive.ubuntu.com:http: Fetched 9248 kB in 30s (304 kB/s)

[stderr] buntu3.21_amd64.deb Unable to connect to azure.archive.ubuntu.com:http: E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/libudev1_245.4-4ubuntu3.21_amd64.deb Unable to connect to azure.archive.ubuntu.com:http: E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd-sysv_245.4-4ubuntu3.21_amd64.deb Unable to connect to azure.archive.ubuntu.com:http:

E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/l/linux/linux-tools-common_5.4.0-147.164_all.deb Unable to connect to azure.archive.ubuntu.com:http: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?`

could you please run apt update firstly?

qaiserali commented 1 year ago

This is also impacting us. Any fix for it?

is still on-going? please copy the error message

We are getting the following error Unable to connect to azure.archive.ubuntu.com:http: Err:19 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 xxd amd64 2:8.1.2269-1ubuntu5.13 Unable to connect to azure.archive.ubuntu.com:http: Err:20 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 vim-common all 2:8.1.2269-1ubuntu5.13 Unable to connect to azure.archive.ubuntu.com:http: Err:26 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-cloud-tools-common all 5.4.0-147.164 Unable to connect to azure.archive.ubuntu.com:http: Err:27 http://azure.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-tools-common all 5.4.0-147.164 Unable to connect to azure.archive.ubuntu.com:http: Fetched 9248 kB in 30s (304 kB/s) [stderr] buntu3.21_amd64.deb Unable to connect to azure.archive.ubuntu.com:http: E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/libudev1_245.4-4ubuntu3.21_amd64.deb Unable to connect to azure.archive.ubuntu.com:http: E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/systemd-sysv_245.4-4ubuntu3.21_amd64.deb Unable to connect to azure.archive.ubuntu.com:http: E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/l/linux/linux-tools-common_5.4.0-147.164_all.deb Unable to connect to azure.archive.ubuntu.com:http: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

could you please run apt update firstly?

I already run apt update and it seems that has fixed the issue.

in-fke commented 1 year ago

Unstable again!? Using ubuntu-latest

W: Tried to start delayed item http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libavcodec58 amd64 7:4.4.2-0ubuntu0.22.04.1, but failed 
W: Tried to start delayed item http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libavformat58 amd64 7:4.4.2-0ubuntu0.22.04.1, but failed 
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/x/x264/libx264-163_0.163.3060%2bgit5db6aa6-2build1_amd64.deb 503 Service Unavailable [IP: 20.54.144.51 80] 
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/x/xvidcore/libxvidcore4_1.3.7-1_amd64.deb 503 Service Unavailable [IP: 20.54.144.51 80] 
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/z/zvbi/libzvbi-common_0.2.35-19_all.deb 503 Service Unavailable [IP: 20.54.144.51 80] 
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/z/zvbi/libzvbi0_0.2.35-19_amd64.deb 503 Service Unavailable [IP: 20.54.144.51 80]
amotl commented 1 year ago

One of our jobs also just tripped over for the same reasons, but it is succeeding now again.

E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/j/java-common/default-jre-headless_1.11-72build2_amd64.deb  Connection failed [IP: 52.252.75.106 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/j/junit4/junit4_4.13.2-1_all.deb  Connection failed [IP: 52.252.75.106 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/libx/libxalan2-java/libxalan2-java_2.7.2-4_all.deb  503  Service Unavailable [IP: 52.252.75.106 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/a/apache-pom/libapache-pom-java_18-1_all.deb  503  Service Unavailable [IP: 52.252.75.106 80]
[...]

-- https://github.com/crate/crate-docs-theme/actions/runs/4982713337/jobs/8918772197#step:4:244

lextoumbourou commented 1 year ago

Down again.

E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/libv/libva/va-driver-all_2.14.0-1_amd64.deb  Unable to connect to azure.archive.ubuntu.com:http:
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/libv/libvdpau/vdpau-driver-all_1.4-3build2_amd64.deb  Unable to connect to azure.archive.ubuntu.com:http:
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/p/pocketsphinx/pocketsphinx-en-us_0.8.0%2breal5prealpha%2b1-14ubuntu1_all.deb  Unable to connect to azure.archive.ubuntu.com:http:
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Quite a show-stopper when this happens. We have apt-get update --fix-missing running in CI, but it doesn't seem to fix it.

LiliDeng commented 1 year ago

Down again.

E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/libv/libva/va-driver-all_2.14.0-1_amd64.deb  Unable to connect to azure.archive.ubuntu.com:http:
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/libv/libvdpau/vdpau-driver-all_1.4-3build2_amd64.deb  Unable to connect to azure.archive.ubuntu.com:http:
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/p/pocketsphinx/pocketsphinx-en-us_0.8.0%2breal5prealpha%2b1-14ubuntu1_all.deb  Unable to connect to azure.archive.ubuntu.com:http:
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Quite a show-stopper when this happens. We have apt-get update --fix-missing running in CI, but it doesn't seem to fix it.

let me create ubuntu case ...

lextoumbourou commented 1 year ago

@LiliDeng can you tell me more about that? How do you create a Ubuntu case?

LiliDeng commented 1 year ago

or try with --fix-missing?

firstly, you need a ubuntu account, then you can create case here https://portal.support.canonical.com/ua/s/case/5008e00000EFPVNAA5/unable-to-connect-to-azurearchiveubuntucomhttp-ip-4011946219-80