Closed fwilhe closed 6 months ago
Hey @fwilhe, We are looking into this
Thanks @shamil-mubarakshin
I think I have even more evidence that it broke in that specific image update.
In this job run it seems like different image versions were mixed randomly?
Looking at a few failed and a few successful jobs, the pattern is clear:
Failed:
Image Version: 20240225.1.0 Job Name: build / images (arm64, metal_secureboot_readonly)
Image Version: 20240225.1.0 Job Name: build / images (arm64, metal_secureboot_readonly_persistence)
Worked:
Image Version: 20240218.1.0 Job Name: build / images (amd64, metal)
Image Version: 20240218.1.0 Job Name: build / images (amd64, kvm_secureboot)
Is there a way to force a specific image version in the runs-on
stanza?
Let me know if I can help with this further.
@fwilhe, thanks for the provided blogpost link, looks like following commit has made it to 6.5.0-1015-azure kernel. crun
is indeed installed as a podman
dependency from official ubuntu repo and there is no backport yet.
Could you try adding following workaround:
- name: patch crun
shell: bash
run: |
curl -Lo ./crun https://github.com/containers/crun/releases/download/1.14.3/crun-1.14.3-linux-amd64
sudo install crun /usr/bin/crun
Unfortunately, image version cannot be selected in runs-on
. Self-hosted runners can be used to run workflows with custom software/hardware.
I was hoping we could avoid manually messing with the installed components, as also the podman version is quite old..
If I understand this correctly, there is no way to use podman build
successfully on ubuntu 22.04 with a 6.5 linux kernel, right? That's a bummer.
Is there any chance that podman and crun will be updated in the ubuntu 22.04 image? I understand that you'd like to avoid to use it from non-official repos, but for the time being that sounds like it just won't work.
Is there any chance that podman and crun will be updated in the ubuntu 22.04 image? I understand that you'd like to avoid to use it from non-official repos, but for the time being that sounds like it just won't work.
I am affected too but I am asking for another solution: Is there any chance that this incompatible kernel is rolled back? The logic would be like that: incompatible kernel caused this issue, obvious fix for this is reverting. Seriously.
We can not roll the kernel back, unfortunately
We can not roll the kernel back, unfortunately
Too bad. Any updates on an updated version of crun and/or podman in that image that fixes the issue?
@fwilhe, podman
, buildah
, skopeo
with their dependencies are installed using apt-get
from official ubuntu repo. At the moment there are no plans on changing the installation method. You could contact ubuntu to get updates to 22.04. In the meantime please use workaround.
One thought: as an alternative workaround you could try using runc instead of crun
$ apt install runc
$ podman run -ti --runtime runc systemd-container
Hi, I am also impacted by this, see:
I'll give the workaround a try.
Hi, thanks for opening the bug in cloud-images. I'll get podman
and crun
added to the bug as well. If i'm understanding all this correctly, there is a change in the 6.5 kernel which causes a break in podman
, specifically with it's dependency on crun
, and a fix is needed.
podman
and crun
are both universe packages. Universe packages are considered to have "community" support, meaning the majority of work comes from the upstream packaging in Debian. looks like the version sync'd from Debian at Ubuntu 22.04 release (3.4.4
) isn't available in Debian anymore, and based on the versioning, i'm going to guess that it already has some patches applied. crun
Ubuntu has the same version as Bullseye, so it may be worth seeing if this is reproducible in Debian Bullseye. If so, we could see about getting an upstream patch applied.
Let me get try and get the bug assigned to the correct packages, and see if there is anyone that can help out.
quick update -- follow on the launchpad bug: https://bugs.launchpad.net/cloud-images/+bug/2056442
we're already working on it. and thanks to @fauust for opening the bug. For anyone following, if you ever hit something in a cloud-based Ubuntu image on one of the big public clouds, feel free to open a bug in https://bugs.launchpad.net/cloud-images . I promise we check, and will try to get things routed to the correct places if it's not an image problem :)
Thanks @j5awry , @fauust
Much appreciated
~podman run --systemd always
seems to be another workaround.~
Apparently not, see below.
BTW, the fix is on it's way, see: https://bugs.launchpad.net/cloud-images/+bug/2056442
podman run --systemd always
seems to be another workaround.
One thought: as an alternative workaround you could try using runc instead of crun
$ apt install runc $ podman run -ti --runtime runc systemd-container
This solution works though!
As the new 24.04 image is now in beta this is the way forward I guess (maybe after the image is not beta anymore)
Description
I think there might be a regression in this release of the ubuntu-22.04 image which breaks podman.
The image updated the kernel from 6.2.y to 6.5.y, but podman/crun don't seem to be updated.
Our build fails with this error link to run:
This is with this image version:
Trying to reproduce, it seems like I am only able to get this image version
20240218.1.0
where the issue does not appear.Tried to reproduce in this repo, but I'm not able to get this with image version
20240225.1.0
.Is this a known issue and version
20240225.1.0
is not in use anymore?This blog post seems to suggest that the crun version is too old.
Platforms affected
Runner images affected
Image version and build link
Fails with:
20240225.1.0
Works with:20240218.1.0
Failing build: https://github.com/gardenlinux/gardenlinux/actions/runs/8061893233/job/22020522535
Is it regression?
yes
Expected behavior
Podman build should work
Actual behavior
Podman build fails with message:
Repro steps
Create a workflow running on ubuntu 22.04 and execute those steps: