search

actions / runner-images

GitHub Actions runner images
MIT License
10.26k stars 3.08k forks source link

Github actions fails after new VMs deployed #9733

Closed eenagy closed 7 months ago

eenagy commented 7 months ago

Description

When running github actions when the VMs are in active deployment the github actions fail. Taken from workflow actions log. Please disable action running, while VM is deploying or only allow queuing. 

 Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [142 B]
Hit:2 http://azure.archive.ubuntu.com/ubuntu jammy InRelease
Get:3 http://azure.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:4 http://azure.archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
Get:5 http://azure.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
Err:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Get:7 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1610 kB]
Get:8 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB]
Get:9 http://azure.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1830 kB]
Get:10 http://azure.archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB]
Get:11 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1070 kB]
Get:12 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB]
Get:13 http://azure.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB]
Get:14 http://azure.archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB]
Hit:15 https://ppa.launchpadcontent.net/ubuntu-toolchain-r/test/ubuntu jammy InRelease
Get:16 http://azure.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1392 kB]
Get:17 http://azure.archive.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB]
Get:18 http://azure.archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1768 kB]
Get:19 http://azure.archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [299 kB]
Get:20 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB]
Get:21 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB]
Reading package lists...
E: Failed to fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: The repository 'https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease' is no longer signed.

Platforms affected

Runner images affected

Image version and build link

Ubuntu LTS

Is it regression?

Not sure

Expected behavior

Either do not run the GitHub workflow or make it queue.

Actual behavior

Workflows fail.

Repro steps

  1. Create a workflow actions
  2. Use Ubuntu LTS
  3. run apt update as workflow step

Temporary Workaround

Actually

sudo apt-get update || true

won't help because apt index would be incomplete and the error to be ignored.

The problem is not in the image itself but in the Microsoft's apt repo, we are reaching them out.

the better workaround would be the following snippet of code:

    run: |
     sudo rm /etc/apt/sources.list.d/microsoft-prod.list
     sudo apt-get update

Update 13:10 UTC issue has been reported fixed

We are keeping an eye

greenstatic commented 7 months ago

Can confirm, I am getting the same error on my workflow runs.

n-thumann commented 7 months ago

Seems to be the same as reported in https://github.com/microsoft/linux-package-repositories/issues/130.

jserpapinto commented 7 months ago

We are getting this as well on all our repos.

Antse commented 7 months ago

same for us

mihaTrajbaric commented 7 months ago

Same issue on our repos.

roharon commented 7 months ago

We got same issues in Github Actions that using ubuntu-latest.

reachlin commented 7 months ago

content-length: 0

(base) lincai@Charles-Cai-MBP16 makomo % curl -v https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease
*   Trying 13.107.213.74:443...
* Connected to packages.microsoft.com (13.107.213.74) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=packages.microsoft.com
*  start date: Feb 20 15:27:19 2024 GMT
*  expire date: Feb 14 15:27:19 2025 GMT
*  subjectAltName: host "packages.microsoft.com" matched cert's "packages.microsoft.com"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure RSA TLS Issuing CA 08
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: packages.microsoft.com]
* [HTTP/2] [1] [:path: /ubuntu/22.04/prod/dists/jammy/InRelease]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET /ubuntu/22.04/prod/dists/jammy/InRelease HTTP/2
> Host: packages.microsoft.com
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/2 200 
< date: Wed, 24 Apr 2024 09:47:50 GMT
< content-length: 0
< x-ms-request-id: 7d5ed39a-501e-0024-4d26-963923000000
< x-ms-version: 2021-08-06
< x-ms-error-code: InvalidRange
< strict-transport-security: max-age=31536000; includeSubDomains
< x-azure-ref: 20240424T094750Z-1656cf4dd4dfpwdca5zpavt7c4000000087g000000000pn9
< cache-control: public, max-age=31536000
< x-fd-int-roxy-purgeid: 66945124
< x-cache: TCP_HIT
< accept-ranges: bytes
< 
* Connection #0 to host packages.microsoft.com left intact
sciutand commented 7 months ago

same here on ubuntu latest

zviryatko commented 7 months ago

For myself added sudo rm /etc/apt/sources.list.d/microsoft-prod.list before the sudo apt-get update I don't care for ms repos in my pipelines, so should work.

JesusPoderoso commented 7 months ago

Some workarounds we've seen are adding || true to the apt update command: sudo apt-get update || true Hope it helps

yongsk0066 commented 7 months ago

same here too 🥹

pavanm-huma commented 7 months ago

Same issue for us too. Changing 'runs-on: latest' to 'runs-on: ubuntu-20.04' helped.

mikhailkoliada commented 7 months ago

Actually

sudo apt-get update || true

won't help because apt index would be incomplete and the error to be ignored.

The problem is not in the image itself but in the Microsoft's apt repo, we are reaching them out.

the better workaround would be the following snippet of code:


        run: |
         sudo rm /etc/apt/sources.list.d/microsoft-prod.list
         sudo apt-get update
sciutand commented 7 months ago

strange my browser can fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease no problem, but with curl it fails. Even when using copy with curl to ensure the headers are the same. ... wget https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease works as well.

dannystaple commented 7 months ago

strange my browser can fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease no problem, but with curl it fails. Even when using copy with curl to ensure the headers are the same. ... wget https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease works as well.

It's 0 bytes, which seems like the main problem.

miya10kei commented 7 months ago

same issue for us.

leovct commented 7 months ago

+1

LiorBaraban commented 7 months ago

Same for us on Ubuntu latest (22.04) during the execution of playwright install --with-deps (since --with-deps triggers an apt-update in the background)

sciutand commented 7 months ago

strange my browser can fetch https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease no problem, but with curl it fails. Even when using copy with curl to ensure the headers are the same. ... wget https://packages.microsoft.com/ubuntu/22.04/prod/dists/jammy/InRelease works as well.

It's 0 bytes, which seems like the main problem.

Screenshot 2024-04-24 at 10 31 36 PM

@dannystaple Not in he browser. Nor wget. For some reason when curl makes the request the origin server refuses to send data. There is also this in the response headers x-ms-error-code: InvalidRange Although I've no idea what http client does apt-get uses, I find that behaviour strange.

omid commented 7 months ago

@sciutand most probably it's because of browser cache. 🤷🏼‍♂️ Try private window. For me content-length is zero! Anyway, this solution is the best for me, since we don't need any of the MS packages (the repo contains dotnet and some more stuff)

beaylott commented 7 months ago

+1

euanhunteratom commented 7 months ago

Same for us on Ubuntu latest (22.04) during the execution of playwright install --with-deps (since --with-deps triggers an apt-update in the background)

Same for us. Running the rm part of https://github.com/actions/runner-images/issues/9733#issuecomment-2074590278 before the playwright install fixed it for us.

xrodriguezd commented 7 months ago

+1

zettlrobert commented 7 months ago

+1

sjagoe commented 7 months ago

Our self-hosted runners (with ephemeral filesystems, so the runner is reset between every job) update apt cache on startup in the install process of the actions runner, so this issue has completely hosed our self-hosted CI runners :grimacing:

I'd have to build new runner host images in order to work around this.

mikhailkoliada commented 7 months ago

Update 13:10 UTC issue has been reported fixed

We are keeping an eye

rvecmanis commented 7 months ago

Seems to be fixed, our runners work as intended.

pntgoswami18 commented 7 months ago

Our github actions are now working as intended, the issue seems fixed for us

mikhailkoliada commented 7 months ago

We got confirmation that the issue has been fully addressed, closing now!