Zypper does not protect pinned kernel versions from removal

[why-not-try-calmer]

As of zypper's latest version, the multiversion setting does not seem to be "weighted", i..e if the user has for example: multiversion.kernels = latest,latest-1,5.13.12-2-default,running and their /boot is running out of space, zypper will not protect the pinned version from removalpurge. So the end user who does sudo zypper dup (with no lock on kernel-* packages) might end up installing new kernels that will take precedence over the pinned one.

One could object that the user is not supposed to run out of space on their /boot, but a nice and friendly zypper would not need to punish the user for this.

Please amend zypper's purge business logic to have pinned version take precedence over all other descriptors in the multiversion list, on account of the fact that if the user chooses to pin a version, that's because they don't want to use any other (until they have unpinned it).

[bzeller]

zypper purge-kernels and zypper dup are 2 different things. Are you having a issue with zypper purge-kernels removing a kernel it should not, or do you have a issue with zypper dup removing some kernels. Also /boot partition size is not taken into account by purge-kernels so this can not be the issue here. It would be helpful to get the zypper log for the command that removed the kernel we are talking about here.

[why-not-try-calmer]

zypper purge-kernels and zypper dup are 2 different things. Are you having a issue with zypper purge-kernels removing a kernel it should not, or do you have a issue with zypper dup removing some kernels. Also /boot partition size is not taken into account by purge-kernels so this can not be the issue here. It would be helpful to get the zypper log for the command that removed the kernel we are talking about here.

Hi Benjamin, thanks for your reply. I have an issue with zypper dup removing a kernel explicitely pinned in a functional multiversion configuration. Can you confirm that there is nothing in the business logic protecting pinned versions from removal when zypper dup is evaluated? A reasonable protection would be: "leave the pinned version alone and remove something else if you have to remove a kernel".

[why-not-try-calmer]

Sorry for using the term "purge" in my originl post; I think I should have been less presuppositional and used the word "removal" instead since this issue is not essentially tied to the zypper purge utility.

[mlandres]

@why-not-try-calmer zypper dup, if multiversion is on, will not remove kernel packages unless dependencies of other packages request this or the user selects this as part of a dependency resolution issue. There are no low disk space or similar conditions which would select packages to remove in order to gain disk space.

The configuration in multiversion.kernels is solely used by the purge-kernels command to decide which kernel to keep. This has no influence on the dependency resolution of any install command. In other words, this does not apply any lock on the kernel packages.

[mlandres]

We can check why the kernel was removed in dup: If 'zypper dup' was used for the update, there should be a solver-testcase at /var/log/updateTestcase-YYYY-MM-DD-hh-mm-ss (date and time of the upadte). Please pack the testcase directory together with the/var/log/zypper.log and attach it to the bugreport (or send it per mail to me or Benjamin...)

[why-not-try-calmer]

zypper_read.log last_2_testcases.zip I hope this will shed light on my story (removed pinned multiversion kernel 5.13-12-2.1)

[mlandres]

@why-not-try-calmer it looks like the dup tescase didn' t touch kernel-default-5.13-12-2.1, nor did the purge-kernels, BUT here kernel-default-5.13-12-2.1 was also the running kernel. The following command will extract the commands that deleted a kernel-default from your /var/log/zypp/history:

sed -n '/|command|/{h};/|remove .kernel-default/{H;x;p}' /var/log/zypp/history

We suspect that actually a purge-kernels command removed it.

The reason would be your spec saying ,5.13.12-2-default,. This is the string uname -r tells, but the code expects the kernel packages version which would be ,5.13.12-2.1, So with the current spec purge-kernels removes the kernel, if it is not also the running one.

[mlandres]

This uname string however is also what zypper purge-kernels --dry-run tells as Running kernel release:. Benjamin is about to fix the code to allow this uname format as well.

[why-not-try-calmer]

sed -n '/|command|/{h};/|remove .kernel-default/{H;x;p}' /var/log/zypp/history

returns:

sudo sed -n '/|command|/{h};/|remove .kernel-default/{H;x;p}' /var/log/zypp/history 2021-06-07 10:31:03|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-06-07 10:31:09|remove |kernel-default|5.12.4-1.1|x86_64|root@localhost| 2021-06-17 09:21:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-06-17 09:21:10|remove |kernel-default|5.12.4-2.1|x86_64|root@localhost| 2021-04-07 02:00:01|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-04-07 02:00:08|remove |kernel-default|5.12.9-1.1|x86_64|root@localhost| 2021-06-30 01:20:58|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-06-30 01:21:06|remove |kernel-default|5.12.10-1.1|x86_64|root@localhost| 2021-07-09 11:14:04|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-07-09 11:14:13|remove |kernel-default|5.12.12-1.1|x86_64|root@localhost| 2021-07-10 01:52:28|command|root@localhost.localdomain|'zypper' 'dup'| 2021-07-10 01:52:48|remove |kernel-default|5.13.0-1.1|x86_64|| 2021-07-14 16:21:45|command|root@localhost.localdomain|'zypper' 'dup'| 2021-07-14 16:21:55|remove |kernel-default|5.13.0-1.2|x86_64|| 2021-07-01 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-07-01 02:00:09|remove |kernel-default|5.12.13-1.1|x86_64|root@localhost| 2021-07-01 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-07-01 02:00:10|remove |kernel-default|5.13.0-1.3|x86_64|root@localhost| 2021-07-08 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-07-08 02:00:10|remove |kernel-default|5.13.1-1.1|x86_64|root@localhost| 2021-07-26 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-07-26 02:00:09|remove |kernel-default|5.13.2-1.1|x86_64|root@localhost| 2021-08-08 08:34:08|command|root@localhost.localdomain|'zypper' 'dup'| 2021-08-08 08:34:16|remove |kernel-default|5.13.6-1.1|x86_64|| 2021-07-29 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-07-29 02:00:09|remove |kernel-default|5.13.4-1.2|x86_64|root@localhost| 2021-07-29 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-07-29 02:00:09|remove |kernel-default|5.13.6-1.2|x86_64|root@localhost| 2021-07-29 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-07-29 02:00:10|remove |kernel-default|5.13.8-1.1|x86_64|root@localhost| 2021-09-01 13:06:22|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-09-01 13:06:29|remove |kernel-default|5.13.12-1.2|x86_64|root@localhost| 2021-09-01 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-09-01 02:00:10|remove |kernel-default|5.13.12-2.1|x86_64|root@localhost| 2021-09-01 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-09-01 02:00:09|remove |kernel-default|5.13.13-1.1|x86_64|root@localhost| 2021-09-08 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-09-08 02:00:10|remove |kernel-default|5.14.0-1.4|x86_64|root@localhost.localdomain| 2021-09-20 18:21:37|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-09-20 18:22:15|remove |kernel-default|5.13.12-2.1|x86_64|root@wifi-unifr-156-220.unifr.ch| 2021-09-20 21:54:03|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-09-20 21:54:11|remove |kernel-default|5.13.12-2.1|x86_64|root@localhost| 2021-09-28 00:12:47|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-09-28 00:12:58|remove |kernel-default|5.13.12-2.1|x86_64|root@localhost.localdomain| 2021-09-28 00:12:58|remove |kernel-default|5.13.12-2.1|x86_64|root@localhost.localdomain| 2021-09-28 00:13:04|remove |kernel-default|5.14.1-1.4|x86_64|root@localhost.localdomain| 2021-09-28 10:26:43|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-09-28 10:26:54|remove |kernel-default|5.13.12-2.1|x86_64|root@localhost.localdomain| 2021-09-28 10:26:54|remove |kernel-default|5.13.12-2.1|x86_64|root@localhost.localdomain| 2021-09-28 10:26:59|remove |kernel-default|5.14.1-1.4|x86_64|root@localhost.localdomain| 2021-10-17 16:06:17|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-10-17 18:07:01|remove |kernel-default|5.14.2-1.3|x86_64|root@localhost.localdomain| 2021-09-29 02:00:02|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-09-29 02:00:11|remove |kernel-default|5.13.12-2.1|x86_64|root@localhost| 2021-10-19 07:41:48|command|root@localhost|'/usr/bin/zypper' '-n' 'purge-kernels'| 2021-10-19 07:44:37|remove |kernel-default|5.14.6-1.4|x86_64|root@localhost.localdomain| 2021-10-19 07:44:37|remove |kernel-default|5.14.6-1.4|x86_64|root@localhost.localdomain| 2021-10-19 07:44:42|remove |kernel-default|5.14.6-2.1|x86_64|root@localhost.localdomain|

[why-not-try-calmer]

@mlandres Thank you for the explanation! In hindsight the situations does not necessarily mean that you should change the admissible formats for specifying kernel versions to accomodate the format I used, if there some early detection mechanism triggering an exception and instructing the user about the expected format. Is there a zypper feature to parse key config files and throw this sort of errors? (This is of course another topic but I'd be interested to know because with it I think I would not have had to bother you :)

[mlandres]

Well, it's not really an error. It's fine to have a pattern, that does not actually match any installed kernel.

The reason for fixing the code is that zypper purge-kernels uses this format in it's output. So it's IMO somewhat reasonable to allow this as input as well. Otherwise we'd fix the command to print the package version rather than the uname string.

[why-not-try-calmer]

Well, it's not really an error. It's fine to have a pattern, that does not actually match any installed kernel.

The reason for fixing the code is that zypper purge-kernels uses this format in it's output. So it's IMO somewhat reasonable to allow this as input as well. Otherwise we'd fix the command to print the package version rather than the uname string.

I fully understand and I am grateful for the very quick hotfix. I agree with your reasoning as well.

As regarding my question about a tool for parsing/evaluating essential config files and reporting erroneous entries to the user, any idea? That is something I cherish from my experience with NixOS-land; do we have something that provides this functionality in openSUSE?

[why-not-try-calmer]

To expand a bit on the question from the comment above: my reasoning when opening this issue was something like:

1. Oh nice, I can direct grub to boot on my pinned kernel (5.13) by selecting the corresponding entry.
2. Oh s**t, after zypper-dupping I have lost access to that grub entry.
3. If zypper had had issues with my pinned version spec in zypp.conf, I would not have been able to boot, and therefore (1) would have been false.
4. Therefore the problem is not with my pinned version spec in zypp.conf

The surprising thing is that (3) is false: the spec was somehow invalid, and yet (1) is true: I was able to boot with the corresponding kernel. Until I couldn't anymore.

So instead of treating the symptom -- expanding the range of valid spec via a pattern -- it would be really good to allow the user to detect invalid zypp.conf settings. Because no matter how tolerant the pattern gets, realistically there will be always a user doing something funny in their zypper settings. There a validator would come in handy.

[why-not-try-calmer]

This was closed too early, I think. Which information shall I report back to the documentation team instructing the user what pattern they should use?

[mlandres]

(Yes, gits auto-close feature is sometimes hasty)

The zypp.conf comment as well as the zypper man page say the the version. This is basically the package version (shown e.g. in zypper se -xs kernel-default).
The kernels source version (the package version without the (appended or embedded) buildcounter) can be used as well. This is the leading part of the uname -r output (before a 2nd '-') the running kernel shows. These two versions are visible e.g in zypper se -xsv --provides kernel-default output after the '=':

i+ | kernel-default | package | 5.3.18-lp152.4.1.gf1d966f | x86_64 | tiwai
    provides: kernel-default = 5.3.18-lp152.4.1.gf1d966f
    provides: kernel-default = 5.3.18-lp152.4.gf1d966f
i+ | kernel-default | package | 5.3.18-lp152.60.1         | x86_64 | repo-update
    provides: kernel-default = 5.3.18-lp152.60
    provides: kernel-default = 5.3.18-lp152.60.1

Documentation should IMO still refer to the version as this works with all libzypp versions.

Maybe also worth mentioning, that this config option solely defines which kernel should be omitted from being cleaned up by the purge-kernels boot-time service or zypper command. It is no general protection like a package lock (see zypper locks) would be.

Regarding the error detection, we'll try to let zypper purge-kernels --dry-run also print the list of installed kernel packages together with the matching the specs. So you can check whether the result will match your expectation.

[why-not-try-calmer]

Thank you very much, I'll link the documentation capturing these instructions in this Issue in a couple of days so that you can confirm that I've correctly understood.

In the mean time you for the quick reaction and the fix!