qubes-dom0-update does not initiate dnf install (correct download, but no install), only affecting some Qubes systems

Aekez commented 6 years ago

Qubes OS version:

Qubes 4.0.

Affected component(s):

dom0
- qubes-dom0-update process (correctly downloads dom0 updates).
- To be exact, "dnf install /var/lib/qubes/updates/rpm/*.rpm" logic is not initiated.

Steps to reproduce the behavior:

This only affects some Qubes systems, i.e. other Qubes systems in the same building does not have this issue.
The exact trigger to this behavior is unknown, but once it triggers its permanent for the Qubes system in question.
It happens 100% of the time (every time new dom0 updates are available) on the affected Qubes system, but it happens 0% of the time on the other un-affected Qubes systems.
I've only seen one Qubes system with this problem so far.
I'm speculating maybe re-installing Qubes on this hardware might fix it, but it might be a good idea to report this anyway, just in case.

Expected behavior:

qubes-dom0-update to automatically verify and install the otherwise correctly downloaded dom0 updates.

Actual behavior:

qubes-dom0-update downloads the updates correctly, but does not install them. This bug is silent, it is easy for the user to overlook that there were updates if they do not pay attention to the download, and realize the fact that no install happens after downloading.
It is uncertain if the hash/key verification happens either, but running rpm -K /var/lib/qubes/updates/rpm/*.rpm shows the downloaded files are ok "whole and matching".

General notes:

Manual fix is easy though, but only if the user realizes the problem and the fix

The manual fix
- Maybe redundant, but using just to be safe rpm -K /var/lib/qubes/updates/rpm/*.rpm
- Then sudo dnf install /var/lib/qubes/updates/rpm/*.rpm
- Everything presumably keeps working normally.
- Except the affected system means every new dom0 update will have the same problem, therefore the user needs to stay awake when running dom0 updates, and be sure not to overlook any downloaded dom0 updates, and then install them manually.

A potential critical problem

The fact the user might not realize the downloaded dom0 updates were not installed, might make this issue more common than realized, and this could also potentially (speculation) be a host to some of the other dom0 issues reported.

Related issues:

marmarek commented 6 years ago

Can you paste example output from sudo qubes-dom0-update from affected system? Does qubes-dom0-update --clean help?

Aekez commented 6 years ago

Certainly, as requested I did it with today's updates' from current-testing. As requested I did without --clean first, followed up and did the exact same steps again but with added --clean option. I'll wait with manually installing these updates for 48 hours in case more information is needed, or wait longer if requested to do so.

Log without --clean option
- Log uploaded here qubes-dom0-update.log
- sudo qubes-dom0-update
- qubes-dom0-update final line reports "No updates available".
- However doing ls /var/lib/qubes/update/rpm shows the updates are indeed available.
- Further doing rpm -K /var/lib/qubes/update/rpm/*.rpm to check hash/keys, looks ok.
- Everything above is included in the uploaded log.

Log with the --clean option
- Log uploaded here qubes-dom0-update-with--clean.log
- sudo qubes-dom0-update --clean
- qubes-dom0-update final line reports "No updates available".
- However doing ls /var/lib/qubes/update/rpm shows the updates are indeed available.
- Further doing rpm -K /var/lib/qubes/update/rpm/*.rpm to check hash/keys, looks ok.
- Everything above is included in the uploaded log.

marmarek commented 6 years ago

Are you sure you don't have those versions already installed? For me it looks like it downloads packages which you already have. Check for example rpm -q xen.

Aekez commented 6 years ago

screenshot_2018-07-17_15-48-41

Note the user is different because I originally replaced it with "user" due for anonymity, but I thought a screenshot might be better, hence the difference.

This bug happens every time there are new dom0 updates available for at least some weeks now. It never made any difference if I use --clean, --refresh or without, it still won't trigger the dnf install process. The downloading of updates to dom0 always seem to work perfectly though.

Aekez commented 6 years ago

I just remembered there is one difference for this machine compared to other Qubes machines that might be important to mention, since it has an integrated Ryzen mobile GPU, installing Qubes normally was not possible (no-graphics) due to kernel version the Linux-firmware version included in the Qubes installer.

I used the unix DD command to disc-clone everything on a different Qubes machine. First I updated everything on the working hardware, and installed the newest available kernel-latest and Linux-firmware in the Qubes repo's at the time. Then I moved everything with DD to the Ryzen laptop then made the system work.

So the act of using DD, could maybe change the dom0 integrity? I do not remember if these two overlap in time though, but I thought it might be important to mention just in case my scenario qubes-dom0-update issue is unique, and if that is indeed the case then this can be closed and I can just wait for the newest Qubes installer (I don't mind at all doing manual installs until then).

marmarek commented 6 years ago

I don't think dd could cause this problem.

Lets see what cause qubes-dom0-update think there are no updates, try this: sudo bash -x /usr/bin/qubes-dom0-update

Aekez commented 6 years ago

I don't think dd could cause this problem.

That's reassuring, it seems like it would have been a wide area of uncertainty if it could.

Here's the log when running sudo bash -x /usr/bin/qubes-dom0-update qubes-dom0-update-with-bash-x.log

strugee commented 6 years ago

I've been poking at this since I'm hit by this bug too. It appears that this check:

https://github.com/QubesOS/qubes-core-admin-linux/blob/9a039f07531ec9621bbe806bad77b0caf96caf26/dom0-updates/qubes-dom0-update#L216

fails. Where is that file supposed to come from?

marmarek commented 6 years ago

On Fri, Aug 17, 2018 at 10:26:54PM -0700, AJ Jordan wrote:

I've been poking at this since I'm hit by this bug too. It appears that this check:

https://github.com/QubesOS/qubes-core-admin-linux/blob/9a039f07531ec9621bbe806bad77b0caf96caf26/dom0-updates/qubes-dom0-update#L216

fails. Where is that file supposed to come from?

It is created by createrepo call here

strugee commented 6 years ago

It is created by createrepo call here

Interesting. I do not have a /usr/bin/createrepo_c on my system, although I do have /usr/bin/createrepo. So it seems like that call is probably failing and the error is not being handled? Why might that binary be missing?

strugee commented 6 years ago

$ rpm -q createrepo
createrepo-0.10.3-10.fc25.noarch

strugee commented 6 years ago

Running sudo ln -s /usr/bin/createrepo /usr/bin/createrepo_c in dom0 solved this issue for me.

marmarek commented 6 years ago

createrepo_c is in different package: createrepo_c. Apparently package dependencies are wrong.

qubesos-bot commented 6 years ago

Automated announcement from builder-github

The package qubes-core-dom0-linux-4.0.16-1.fc25 has been pushed to the r4.0 testing repository for dom0. To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

qubesos-bot commented 6 years ago

Automated announcement from builder-github

The package qubes-core-dom0-linux-4.0.16-1.fc25 has been pushed to the r4.0 stable repository for dom0. To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

lunarthegrey commented 5 years ago

Hmm I am seeing this bug with qubes-core-dom0-linux-4.0.17-1.fc25 and using the new fedora-29 template as an update VM. I see the same behavior where updates are downloaded but not installed. But when inspecting the packages that were downloaded, I can see those same versions are already installed. Using --clean doesn't make any difference either.

Thoughts @marmarek ?

Edit: I missed the fact that dnf said it was reinstalling the packages. But I am unsure why, as I'm executing sudo qubes-dom0-update --clean --verbose.

QubesOS / qubes-issues