search

actions / runner-images

GitHub Actions runner images
MIT License
9.79k stars 3k forks source link

Podman regression in Ubuntu 22.04 image released Feb 25th? #9425

Closed fwilhe closed 6 months ago

fwilhe commented 6 months ago

Description

I think there might be a regression in this release of the ubuntu-22.04 image which breaks podman.

The image updated the kernel from 6.2.y to 6.5.y, but podman/crun don't seem to be updated.

Our build fails with this error link to run:

STEP 1/1: FROM ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac
Trying to pull ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac...
Getting image source signatures
Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1
Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1
Copying config sha256:1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5
Writing manifest to image destination
Storing signatures
COMMIT localhost/builder
--> 1eba10d0345
Successfully tagged localhost/builder:latest
Successfully tagged ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac
1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5
Error: OCI runtime error: chmod `run/shm`: Operation not supported
Error: Process completed with exit code 126.

This is with this image version:

Current runner version: '2.313.0'
Operating System
  Ubuntu
  22.04.4
  LTS
Runner Image
  Image: ubuntu-22.04
  Version: 20240225.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240225.1/images/ubuntu/Ubuntu2204-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1

Trying to reproduce, it seems like I am only able to get this image version 20240218.1.0 where the issue does not appear.

Tried to reproduce in this repo, but I'm not able to get this with image version 20240225.1.0.

Is this a known issue and version 20240225.1.0 is not in use anymore?

This blog post seems to suggest that the crun version is too old.

Platforms affected

Runner images affected

Image version and build link

Fails with: 20240225.1.0 Works with: 20240218.1.0

Failing build: https://github.com/gardenlinux/gardenlinux/actions/runs/8061893233/job/22020522535

Is it regression?

yes

Expected behavior

Podman build should work

Actual behavior

Podman build fails with message:

Error: OCI runtime error: chmod `run/shm`: Operation not supported

Repro steps

Create a workflow running on ubuntu 22.04 and execute those steps:

    - name: install required packages
      shell: bash
      run: |
        sudo rm -rf /{usr/{local/{lib/{android,heroku},.ghcup,share/{dotnet,powershell,miniconda,swift}},share/{dotnet,miniconda,swift}},opt/{hostedtoolcache,microsoft},imagegeneration}
        sudo docker system prune -a -f
        sudo apt-get update
        sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends qemu-user-static

    - run: |
        git clone https://github.com/gardenlinux/gardenlinux
        cd gardenlinux
        ./build base-amd64 base-arm64
shamil-mubarakshin commented 6 months ago

Hey @fwilhe, We are looking into this

fwilhe commented 6 months ago

Thanks @shamil-mubarakshin

I think I have even more evidence that it broke in that specific image update.

In this job run it seems like different image versions were mixed randomly?

Looking at a few failed and a few successful jobs, the pattern is clear:

Failed:

Image Version: 20240225.1.0 Job Name: build / images (arm64, metal_secureboot_readonly)
Image Version: 20240225.1.0 Job Name: build / images (arm64, metal_secureboot_readonly_persistence)

Worked:

Image Version: 20240218.1.0 Job Name: build / images (amd64, metal)
Image Version: 20240218.1.0 Job Name: build / images (amd64, kvm_secureboot)

Is there a way to force a specific image version in the runs-on stanza?

Let me know if I can help with this further.

shamil-mubarakshin commented 6 months ago

@fwilhe, thanks for the provided blogpost link, looks like following commit has made it to 6.5.0-1015-azure kernel. crun is indeed installed as a podman dependency from official ubuntu repo and there is no backport yet. Could you try adding following workaround:

    - name: patch crun
      shell: bash
      run: |
        curl -Lo ./crun https://github.com/containers/crun/releases/download/1.14.3/crun-1.14.3-linux-amd64
        sudo install crun /usr/bin/crun

Unfortunately, image version cannot be selected in runs-on. Self-hosted runners can be used to run workflows with custom software/hardware.

fwilhe commented 6 months ago

I was hoping we could avoid manually messing with the installed components, as also the podman version is quite old..

If I understand this correctly, there is no way to use podman build successfully on ubuntu 22.04 with a 6.5 linux kernel, right? That's a bummer.

Is there any chance that podman and crun will be updated in the ubuntu 22.04 image? I understand that you'd like to avoid to use it from non-official repos, but for the time being that sounds like it just won't work.

mafalb commented 6 months ago

Is there any chance that podman and crun will be updated in the ubuntu 22.04 image? I understand that you'd like to avoid to use it from non-official repos, but for the time being that sounds like it just won't work.

I am affected too but I am asking for another solution: Is there any chance that this incompatible kernel is rolled back? The logic would be like that: incompatible kernel caused this issue, obvious fix for this is reverting. Seriously.

mikhailkoliada commented 6 months ago

We can not roll the kernel back, unfortunately

fwilhe commented 6 months ago

We can not roll the kernel back, unfortunately

Too bad. Any updates on an updated version of crun and/or podman in that image that fixes the issue?

shamil-mubarakshin commented 6 months ago

@fwilhe, podman, buildah, skopeo with their dependencies are installed using apt-get from official ubuntu repo. At the moment there are no plans on changing the installation method. You could contact ubuntu to get updates to 22.04. In the meantime please use workaround.

mafalb commented 6 months ago

One thought: as an alternative workaround you could try using runc instead of crun

$ apt install runc
$ podman run -ti --runtime runc systemd-container
fauust commented 6 months ago

Hi, I am also impacted by this, see:

I'll give the workaround a try.

fauust commented 6 months ago

Just opened https://bugs.launchpad.net/cloud-images/+bug/2056442

j5awry commented 6 months ago

Hi, thanks for opening the bug in cloud-images. I'll get podman and crun added to the bug as well. If i'm understanding all this correctly, there is a change in the 6.5 kernel which causes a break in podman, specifically with it's dependency on crun, and a fix is needed.

podman and crun are both universe packages. Universe packages are considered to have "community" support, meaning the majority of work comes from the upstream packaging in Debian. looks like the version sync'd from Debian at Ubuntu 22.04 release (3.4.4) isn't available in Debian anymore, and based on the versioning, i'm going to guess that it already has some patches applied. crun Ubuntu has the same version as Bullseye, so it may be worth seeing if this is reproducible in Debian Bullseye. If so, we could see about getting an upstream patch applied.

Let me get try and get the bug assigned to the correct packages, and see if there is anyone that can help out.

j5awry commented 6 months ago

quick update -- follow on the launchpad bug: https://bugs.launchpad.net/cloud-images/+bug/2056442

we're already working on it. and thanks to @fauust for opening the bug. For anyone following, if you ever hit something in a cloud-based Ubuntu image on one of the big public clouds, feel free to open a bug in https://bugs.launchpad.net/cloud-images . I promise we check, and will try to get things routed to the correct places if it's not an image problem :)

fwilhe commented 6 months ago

Thanks @j5awry , @fauust

Much appreciated

fauust commented 6 months ago

~podman run --systemd always seems to be another workaround.~ Apparently not, see below.

BTW, the fix is on it's way, see: https://bugs.launchpad.net/cloud-images/+bug/2056442

dillfrescott commented 5 months ago

podman run --systemd always seems to be another workaround.

Screenshot 2024-03-14 200807

dillfrescott commented 5 months ago

One thought: as an alternative workaround you could try using runc instead of crun

$ apt install runc
$ podman run -ti --runtime runc systemd-container

This solution works though!

fwilhe commented 3 months ago

As the new 24.04 image is now in beta this is the way forward I guess (maybe after the image is not beta anymore)