search

Bleuzen / ubuntu-kernel-autoremove

1 stars 1 forks source link

General discussion #1

Open Bleuzen opened 2 years ago

Bleuzen commented 2 years ago

@mmikowski

Initial testing showed the packages were marked as auto, however, neither unattended upgrades nor sudo apt autoremove -y removed them. We found manual installed of earlier kernels including 3 of the same series added them to /etc/apt/apt.conf.d/01autoremove-kernels under APT::NeverAutoRemove. Test using sudo unattended-upgrades --dry-run --debug

Ubuntu seems to keep some kernels from being autoremoved by default. From /etc/kernel/postinst.d/apt-auto-removal: image

This script generates /etc/apt/apt.conf.d/01autoremove-kernels, which contains the kernel packages that should be kept:

image

At least that is on Linux Mint / Ubuntu 20.04. However something changed here with newer Ubuntu versions. On Kubuntu 22.04, these files look a lot simpler:

image

Idk. yet how this is supposed to work. Only the last installed kernel is saved on 22.04 but still autoremove seems to keep more than that.

@mmikowski which version of *Ubuntu are you using + how many kernels do you have installed for testing?

mmikowski commented 2 years ago

@Bleuzen @eeickmeyer spent the day testing.

Thank you for the great notes above. We are running 20.04 and 22.04, although current development is on 20.04. We are using 2-4 kernels for testing (oem, hwe, earlier versions of these). /boot is the default, 705MiB. See note below about raising this to 2.0GiB.

Analysis of unattended-upgrades in 20.04

  1. It protects the two highest versions of a any installed kernels. If, however, you install a prior version of a kernel (e.g. sudo apt install linux-image-5.14.0-1008-oem) it protects that too. It appears this protection remains in place until the next kernel install. The installed kernel version is received as the first argument in /etc/kernel/postinst.d/apt-auto-removal.
  2. It does not consider or remove kernels and packages where the top-level meta package was removed. For example, sudo apt purge linux-generic-hwe-20.04 does not result in the associated packages being removed even when all of them are marked as auto. This appears as an issue with the autoremove mechanism of apt and in unattended-upgrades. Neither sudo unattended-upgrades --debug --dry-run nor sudo apt autoremove resulted in removal of these files.
  3. It does not adjust for, or warn about, available disk space on /boot
  4. It does not actually protect the latest 2 versions of a particular meta package like linux-generic-hwe-20.04. Instead, it protects up to 4 blessed kernels: latest-version (the first version in kernel list reverse sorted by version), previous version (2nd entry in sorted list), last installed version (received as $1), and running version (uname -r).
  5. It doesn't consider user intent, but uses rigid logic of (4).

Assessment

Our findings show that even if kernel packages are marked auto, deficiencies in the unattended-upgrades and apt autoremove mechanisms result in substantial deficits compared to kernel-cleaner. Most critically, because of its lack of regard for disk space, unattended-upgrades can allow a over-full disk without alerting the user, especially on small /boot partition devices. However, because unattended-upgrades and apt autoremove mechanisms are well-established and have compelling broad infrastructure, they should be the foundation of any long-term and upstream solution.

We should file a bug with 22.04 to bump the /boot partition to 2.0GiB because the Ubuntu 20.04 and 22.04 /boot partition size for encrypted disks is far too small. At just 705M, it can handle only 3 concurrent kernels. We would like to allow for up to 3 top-level kernel series (e.g. hwe, oem, lowlatency) and keep two versions for each. The disk space requirement is therefore 3 x 2 x 200MiB = 1.2GiB; we should like to round up to 2.0GiB for additional safety.

Proposal for Focal (20.04)

At present, most of the shortcomings above have been addressed (1, 2, 3) or can be easily address (4, 5) by our kernel-cleaner script. To deliver benefits to the user most quickly, we propose to update that tool and continue to use it to refine the logic. We have an good set of automated tests to ensure the logic works as expected.

We can check this in here if you wish. We have already made enhancements for issues 2 and 3. Issue 4 is pretty easy to implement, and issue 5 may be better handled in the next proposal and is discussed below.

Proposal for Jammy (22.04)

I hope that is helpful!

mmikowski commented 2 years ago

Hi @Bleuzen!

Have you had a chance to review the above? What do you think? Ultimately, for 22.04 I think unattended-upgrades should do the following:

There have been a couple advancements for 22.04 which should be useful for this effort:

mmikowski commented 2 years ago

@Bleuzen are you still interested in this? I can add the kernel cleaner script for collaboration here as well if you like. Please let me know!

I think this is a very important project which I'd like to see adopted up-stream. Here is a DFMEA assessment which discusses how catastrophic this problem can be for even advanced users.

Bleuzen commented 2 years ago

@Bleuzen are you still interested in this?

Sure, just have limited time and energy currently.

Also note that my situation / motivation for this may be a little different. I don't use unattended-upgrades or oem kernels and only care about 22.04 as I already upgraded our machines here.

I can add the kernel cleaner script for collaboration here as well if you like.

Yes this would be a nice thing to have :)

I think this is a very important project which I'd like to see adopted up-stream.

Agree, things like this should work ootb.

mmikowski commented 2 years ago

Hi @Bleuzen:

Great, I'll add the script and the tests, probably tomorrow. 22.04 is a good target. With some luck, we see the package kit fix soon.

Discover does not clean up any auto installed packages AFAIK. Unattended upgrades does, but it's logic appears a bit mysterious.

If you are maintaining a number of machines, you might want to use the mark-auto script from above, unattended upgrades (default config) and kernel cleaner script until you are comfortable that unattended upgrades works well enough on its own. If you want to discuss that some time, let me know.

Cheers, Mike

mmikowski commented 2 years ago

@Bleuzen Sorry this took a while, but I've added the script and unit tests. I disentangled the code from the common libs we used to keep it as straight-forward as possible. We can bring this back with the GUI as we progress.

One thing we'd really like to add is the ability for users to select the desired kernel flavors (e.g. oem, hwe, lowlatency) and then return only the last two versions of those. This should be fairly easy.