search

QubesOS / qubes-issues

The Qubes OS Project issue tracker
https://www.qubes-os.org/doc/issue-tracking/
541 stars 48 forks source link

Update microcode_ctl package for R3.2 #3703

Closed phagara closed 6 years ago

phagara commented 6 years ago

Qubes OS version:

R3.2

Affected component(s):

Intel CPU microcode package for dom0 (microcode_ctl)

Steps to reproduce the behavior:

Expected behavior:

Actual behavior:

General notes:

Since the Fedora version used in R3.2 dom0 is long since EOL, upstream will not update the microcode_ctl package. Intel recently released a new batch of microcode updates after pulling the previous one due to some bugs. These new microcode files should contain workarounds for some variant(s) of the Spectre/Meltdown vulnerabilities, so it would be nice to have them in R3.2 too (especially since a lot of older hardware will not be getting BIOS updates containing the new microcode).

This would require downloading the new microcode files from Intel and building a new version of microcode_ctl package with them. Perhaps the builds for newer Fedora releases could be useful too.

Related issues:

HW42 commented 6 years ago

For R4.0 (R3.2 WIP):

Attention: The microcode_ctl packages contains non-free code. But we already distribute it through the Fedora package in the installer.

To actually use it we need to pass ucode=scan to Xen. @marmarek: How should we do this? AFAICS currently the grub/efi parameters are set once by the installer ...

3hhh commented 6 years ago

There's 2 ways to load Microcode. [1] The Intel description [3] apparently refers to the legacy one. I'm not sure whether Xen supports the legacy one - it's at least not mentioned in [4].

I think both require certain kernel options to be set in order to work. These are apparently set by Qubes [2].

What is strange however: Neither /sys/devices/system/cpu/microcode/reload nor /dev/cpu/microcode exist in dom0 on my system and sudo journalctl -b0 | grep micro doesn't mention anything with microcode_ctl installed, i.e. we might have a bug here. EDIT: Ah ok that's probably https://github.com/QubesOS/qubes-vmm-xen/pull/33

Fyi: cat /proc/cpuinfo | grep microcode tells you the version you're on.

[1] https://github.com/torvalds/linux/blob/master/Documentation/x86/microcode.txt [2] https://github.com/QubesOS/qubes-linux-kernel/blob/stable-3.18/config [3] https://downloadcenter.intel.com/download/27591/Linux-Processor-Microcode-Data-File [4] https://wiki.xenproject.org/wiki/XenParavirtOps/microcode_update

marmarek commented 6 years ago

To actually use it we need to pass ucode=scan to Xen. @marmarek: How should we do this? AFAICS currently the grub/efi parameters are set once by the installer ...

For grub, we can do similar thing as with automatic USBVM setup: append to /etc/default/grub:

GRUB_CMDLINE_XEN_DEFAULT="$GRUB_CMDLINE_XEN_DEFAULT ucode=scan"

For EFI, we use sed on /boot/efi/EFI/qubes/xen.cfg...

Probably should be part of %postinst on xen-hypervisor. And make sure to not redo it when the setting is already there.

marmarek commented 6 years ago

Probably should be part of %postinst on xen-hypervisor. And make sure to not redo it when the setting is already there.

or microcode_ctl package. According to xen docs, ucode=scan will not work with EFI, that one needs ucode=<filename> in /boot/efi/EFI/qubes/xen.cfg.

marmarek commented 6 years ago

But despite documentation, ucode=scan command line option works with EFI too.

marmarek commented 6 years ago

The current microcode_ctl package breaks installation: https://openqa.qubes-os.org/tests/392#step/install_do_user/8 Testing a fix here: https://github.com/marmarek/qubes-intel-microcode/tree/postinstall-fix

marmarek commented 6 years ago

Above fix works. Also adding ucode=scan done.

HW42 commented 6 years ago

We can avoid using the rather hacky editing of the command line by just changing the default in Xen (you can still disable microcode loading with ucode=0). Didn't push yet because something with the microcode loading is broken with Xen 4.6 (system freeze or crash).

marmarek commented 6 years ago

I'd rather limit number of our patches to Xen, rather than adding new ones. Unless that could be accepted upstream.

3hhh commented 6 years ago

This looks pretty Intel-specific so far.

Was support for AMD ucode updates considered?

phagara commented 6 years ago

@3hhh microcode_ctl package is "obsolete", it's being used only because Intel does not (yet?) provide microcode blobs compatible with Linux kernel ucode loader (AMD apparently does). In Qubes and other RPM-based distros, AMD ucode is provided by the linux-firmware package. Still, Qubes will need to copy the relevant ucode files to /boot and tell Xen about it, exactly like discussed in the comments above.

mirrorway commented 6 years ago

The new microcode breaks suspend on my corebooted Thinkpad (4.0rc5). Instead of waking up from sleep, it restarts.

marmarek commented 6 years ago

@mirrorway I guess its rather xen-4.8.3-4, not microcode itself. See https://github.com/QubesOS/qubes-issues/issues/3738

h01ger commented 6 years ago

On Wed, Mar 28, 2018 at 07:20:09AM -0700, mirrorway wrote:

The new microcode causes my corebooted Thinkpad on 4.0rc5 to restart instead of waking up from sleep.

it would be great if you and the original submitter could state which thinkpad models you are using and also which coreboot version. Thanks!

-- cheers, Holger

mirrorway commented 6 years ago

T530, coreboot+grub from Dec 2017 - CBET4000 4.6-2454-g33cd28e7ac

It seemed like it was caused by the microcode update, which upgraded my ucode to 0x1f. xen-4.8.3-4 resumes fine after I remove ucode=scan (reverting to ucode 0x1b).

Typo: it's grub, not seabios.

marmarek commented 6 years ago

What about xen-4.8.3-3 while keeping ucode=scan?

mirrorway commented 6 years ago

How do I downgrade xen? Installing the downgraded version doesn't work:

$ sudo qubes-dom0-update xen-4.8.3-3.fc25.x86_64
Using sys-whonix as UpdateVM to download updates for Dom0; this may take some time...
qubes-dom0-current-testing                                  | 3.6 kB  00:00     
--> Running transaction check
---> Package xen.x86_64 2001:4.8.3-3.fc25 will be installed
--> Processing Dependency: xen-runtime = 4.8.3-3.fc25 for package: 2001:xen-4.8.3-3.fc25.x86_64
--> Finished Dependency Resolution
xen-4.8.3-3.fc25.x86_64.rpm                                 |  77 kB  00:00     
find: '/var/lib/qubes/dom0-updates/var/cache/yum': No such file or directory
Qubes OS Repository for Dom0                     24 MB/s |  26 kB     00:00    
No package xen-4.8.3-3.fc25.x86_64 available.
Error: Unable to find a match.
phagara commented 6 years ago

Normally, sudo qubes-dom0-update --action=downgrade xen xen-hvm xen-libs xen-runtime (R4 might have different dependencies) would do the trick (you may also specify the full version for each package if just the previous update won't cut it). However, this might not work for you using whonix since I'm getting the following error using a debian-9 based VM:

ERROR: yum version installed in VM updatevm does not suppport --downloadonly option
ERROR: only 'install' and 'upgrade' actions supported (downgrade not)

As a workaround for when you get the above error, cd into /var/lib/qubes/updates/rpm or similar and install manually using yum downgrade xen*.rpm in dom0.

mirrorway commented 6 years ago 
$ ls
xen-4.8.3-3.fc25.x86_64.rpm      xen-libs-4.8.3-3.fc25.x86_64.rpm
xen-hvm-4.8.3-3.fc25.x86_64.rpm  xen-runtime-4.8.3-3.fc25.x86_64.rpm

$ sudo dnf downgrade xen*.rpm
Qubes OS Repository for Dom0                     29 MB/s |  30 kB     00:00    
Error: package python3-xen-2001:4.8.3-4.fc25.x86_64 requires xen-libs = 4.8.3-4.fc25, but none of the providers can be installed.
package xen-runtime-2001:4.8.3-3.fc25.x86_64 requires xen-libs = 4.8.3-3.fc25, but none of the providers can be installed.
package xen-hvm-2001:4.8.3-3.fc25.x86_64 requires xen-libs = 4.8.3-3.fc25, but none of the providers can be installed.
package xen-2001:4.8.3-3.fc25.x86_64 requires xen-runtime = 4.8.3-3.fc25, but none of the providers can be installed
(try to add '--allowerasing' to command line to replace conflicting packages)

$ sudo dnf downgrade --allowerasing xen*.rpm
Qubes OS Repository for Dom0                     29 MB/s |  30 kB     00:00    
Dependencies resolved.
Error: The operation would result in removing the following protected packages: qubes-core-dom0.

Seems like it could be dangerous.

marmarek commented 6 years ago

Actually the most relevant package here is xen-hypervisor. Try downgrading just this one.

mirrorway commented 6 years ago

No resume issues with xen-hypervisor-4.8.3-3.fc25.x86_64 and updated microcode loaded.

$ rpm -qa | grep '^xen'
xen-libs-4.8.3-4.fc25.x86_64
xen-hvm-4.8.3-4.fc25.x86_64
xen-hypervisor-4.8.3-3.fc25.x86_64
xen-hvm-stubdom-linux-1.0.7-1.fc25.x86_64
xen-runtime-4.8.3-4.fc25.x86_64
xen-4.8.3-4.fc25.x86_64
xen-licenses-4.8.3-4.fc25.x86_64

$ rpm -q microcode_ctl
microcode_ctl-2.1-22.qubes2.fc25.x86_64

$ cat /proc/cpuinfo | grep microcode
microcode   : 0x1f
microcode   : 0x1f
microcode   : 0x1f
microcode   : 0x1f
microcode   : 0x1f
microcode   : 0x1f
microcode   : 0x1f
microcode   : 0x1f
HW42 commented 6 years ago

For R3.2: https://github.com/QubesOS/qubes-vmm-xen/pull/36

HW42 commented 6 years ago

For AMD microcode: https://github.com/HW42/qubes-linux-firmware. This also fixes https://github.com/QubesOS/qubes-issues/issues/3796. Of course this contains a lot of non-free code, but like the Intel microcode we already ship an older version of it as part of the installer so this should not be a (new) problem.

@marmarek: I don't have a recent AMD CPU for testing. So unless you happen to be able to test it please upload the linux-firmware package first to unstable so that somebody else can test.

HW42 commented 6 years ago

@marmarek: And you need to upload the files contained in linux-firmware-20180402-83.git8c1e439c.fc26.src.rpm to https://ftp.qubes-os.org/distfiles/.

marmarek commented 6 years ago

linux-firmware package(s) uploaded to unstable repo for R4.0. Anyone with AMD CPU here? If needed, I can upload it also for R3.2. How to test:

sudo qubes-dom0-update --enablerepo=qubes-dom0-unstable linux-firmware

Then reboot and see if things still works. It may be good idea to wait for xen-4.8.3-5 with fix for QubesOS/qubes-issues#3738

HW42 commented 6 years ago

@3hhh: You had asked about AMD. Had you time to test the updated linux-firmware packages (see @marmarek's comment directly above)?

For anybody who has a test machine with a recent AMD CPU it would be helpful if you could update the xen packages from the testing repo and the install the linux-firmware package from the unstable repo (see above). Then boot with an added Xen option loglvl=all and look for the BTI mitigation messages. Also please test suspend/resume.

3hhh commented 6 years ago

Sorry, I don't run an AMD CPU. I had only noticed that all of the commits before were Intel centric.

I just asked for help for this thread on the users ML: https://groups.google.com/forum/#!topic/qubes-users/1XCidVWJ-OI

sergiodamatta commented 6 years ago

I am using: AMD AM3 Phenom II 1100T X6 6 cores 6 threads Asus Sabertooth R2.0 BIOS 2901 990FX DDR3 HVM, IOMMU, SLAT Qubes R4.0, Xen 4.8.3-5, Kernel 4.14.18-1

Normal boot with Xen Boot with ucode=scan log1v1=all, nothing observed

grep -i bti /var/log/xen/console/hypervisor.log shows nothing

3hhh commented 6 years ago

Thanks a lot to you two for testing!

@marmarek @HW42 Please advise on how to proceed.

awokd commented 6 years ago

FWIW these updates don't hurt a corebooted A10-5750M. Microcode isn't getting updated, but I think that's an upstream issue somewhere with the equivalency table (that's originally why I went the route of patching coreboot instead) or fam15 updates aren't yet included. (XEN) BTI mitigations: Thunk LFENCE, Others: RSB_NATIVE RSB_VMEXIT and dom0 kernel: Spectre V2 : Mitigation: Full AMD retpoline showing in the logs.