search

actions / runner-images

GitHub Actions runner images
MIT License
10.19k stars 3.06k forks source link

Installation failure for grub-efi-amd64-signed on ubuntu-latest #7192

Closed sushantmimani closed 1 year ago

sushantmimani commented 1 year ago

Description

sudo apt-get -y upgrade --fix-missing fails with the following error on ubuntu-latest

mount: /var/lib/grub/esp: special device /dev/disk/by-id/scsi-14d534654202020200690df035523a242afc173af9e57f2a4-part15 does not exist.
[122](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:123)
dpkg: error processing package grub-efi-amd64-signed (--configure):
[123](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:124)
 installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32
[124](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:125)
Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
[125](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:126)
Processing triggers for hicolor-icon-theme (0.17-2) ...
[126](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:127)
Processing triggers for man-db (2.10.2-1) ...
[127](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:128)
Errors were encountered while processing:
[128](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:129)
 grub-efi-amd64-signed
[129](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:130)
needrestart is being skipped since dpkg has failed
[130](https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774#step:3:131)
E: Sub-process /usr/bin/dpkg returned an error code (1)

Platforms affected

Runner images affected

Image version and build link

Image: ubuntu-22.04 Version: 20230219.1 https://github.com/godaddy/asherah/actions/runs/4265406600/jobs/7424702774

Is it regression?

No

Expected behavior

Command should execute successfully.

Actual behavior

Command is failing.

I've downgraded to ubuntu-20.04 where this command runs successfully.

Repro steps

  1. Create a workflow that uses ubuntu-latest
  2. Run command sudo apt-get -y upgrade --fix-missing
christophebedard commented 1 year ago

Is it regression?

No

I would argue it is a regression.

ddobranic commented 1 year ago

@sushantmimani hello, thank you for reporting. until the issue is fixed, please use this workaround:

echo RESET grub-efi/install_devices | debconf-communicate grub-pc
mikhailkoliada commented 1 year ago

Hello!

echo 'APT::Get::Always-Include-Phased-Updates "false";' | sudo tee /etc/apt/apt.conf.d/99-phased-updates

Is your friend. We have phased updates enabled on Ubuntu, this usually helps new updates to be delivered faster, but sometimes it also breaks some compatibility / upgrades (if someone runs ones during canonical updates). You may wish to add it to your workflows if you are running updates often on your own. Feel free to ask more questions if you have, I am going to close the ticket as there is nothing actable for us right now.

tothxa commented 1 year ago

@mikhailkoliada Are you sure it's a problem with phased upgrades?

The relevant part of the failure log is:

Setting up grub-efi-amd64-signed (1.187.3~22.04.1+2.06-2ubuntu14.1) ...
mount: /var/lib/grub/esp: special device /dev/disk/by-id/scsi-14d534654202020200690df035523a242afc173af9e57f2a4-part15 does not exist.
dpkg: error processing package grub-efi-amd64-signed (--configure):
 installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32

As far as I can tell, phased upgrades should result in packages being held back, but here grub is getting upgraded (even though I don't think many of the affected workflows will actually ever use the upgraded version). edit: sorry, my mistake, you said to disable phased upgrades, not enable them, but that doesn't change the part that this is a local config error

A changed UUID is a local problem instead, isn't it?

congyiwu commented 1 year ago

Adding notes here b/c I hit this in my dev environment while fixing a different image for Larger Runners.

This smells a lot like https://github.com/canonical/cloud-init/issues/4091, which is currently still open. GitHub Actions runners still boot with BIOS. If I'm correct, this issue could reoccur whenever the grub-efi-amd64-signed package is updated twice within a short period of time, if:

  1. grub-efi-amd64-signed is updated in the package repo
  2. We build the Ubuntu image for GitHub Actions with a base image that doesn't have the new grub-efi-amd64-signed version.
  3. The grub-efi/install_devices debconf key is initially unset
  4. During the image build, we'll upgrade grub-efi-amd64-signed: https://github.com/actions/runner-images/blob/9d1c61329a11dbc41c69dbde7ba7a1596cafe218/images/linux/scripts/base/repos.sh#L16 The logs will probably look like this: 
    Trying to migrate /boot/efi into esp config
Installing grub to /boot/efi.
Installing for x86_64-efi platform.
grub-install: warning: EFI variables cannot be set on this system.
grub-install: warning: You will have to complete the GRUB setup manually.
Installation finished. No error reported.

    This also sets grub-efi/install_devices to a path in /dev/disk/by-id/ based on the current ID of the disk

  5. We release the image
  6. grub-efi-amd64-signed is updated in the package repo again
  7. We create a runner with the image. This resets the disk ID
  8. A customer runs apt-get upgrade They'll get an error like this: 
    Setting up grub-efi-amd64-signed (1.187.3~22.04.1+2.06-2ubuntu14.1) ...
mount: /var/lib/grub/esp: special device /dev/disk/by-id/scsi-14d534654202020200690df035523a242afc173af9e57f2a4-part15 does not exist.
dpkg: error processing package grub-efi-amd64-signed (--configure):
installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32

And it looks this reoccurred last week: https://github.com/orgs/community/discussions/47863#discussioncomment-7212869

From looking at Jammy/updates in https://launchpad.net/ubuntu/+source/grub2-signed/+publishinghistory", the timing sort of lines up: