search

linux-surface / linux-surface

Linux Kernel for Surface Devices
4.68k stars 205 forks source link

Enrolling secureboot keys makes the device freeze on the Microsoft logo after firmware update #1274

Open awptechnologies opened 8 months ago

awptechnologies commented 8 months ago

EDIT by @StollD:

The TL;DR; of this issue: After a recent firmware update, enrolling secureboot certificates causes the device to freeze on the Microsoft logo.

The new firmware ships with a security feature called NX mode. It means that the firmware will not allocate memory that is readable, writable and executable by default. Applications can set a flag in their binary during compilation to indicate whether they are compatible with NX mode, and the UEFI should only turn it on when the applications indicate compatibility.

Unfortunately, Microsoft decided to ignore this flag and are always enabling NX mode.

When you enroll a certificate, a flag in the UEFI variable storage is set that will cause the Shim bootloader to launch a program called "MokManager". Unfortunately, this program is currently incompatible with NX mode, and will cause the firmware to stop executing it right after getting loaded.

Furthermore, when you try to boot an installation image in this state, it is possible that the shim bootloader will abort with an error. This is because it sees the UEFI variable and tries to launch MokManager, but some installation images are not shipping with MokManager.

The good news is: This issue is fixed on the shim main branch, so once the distributions update their shim, this issue should disappear. The bad news is, that it is not possible for us to fix this, since we can't get a signed shim / MokManager from Microsoft.

For now there are three possible solutions:

Some additional notes (IMPORTANT):

Since this issue can be fixed by booting the Mint image once or by disabling secureboot, I would recommend against downgrading the firmware, unless you absolutely want to or you know what you are doing. Feel free to ask for assistance with the process in our support channel on Matrix: https://matrix.to/#/#linux-surface-support:matrix.org

Original post:

After install I get no boot. Locked up on the Microsoft logo

tarsil commented 8 months ago

I'm the same. Infinite loop. I don't even have windows anymore and I'm desperate not knowing what to do anymore. It does not even boot from USB anymore

nickdepinet commented 8 months ago

what surface device and distro are you using, additionally did you attempt to setup secure-boot before this happened?

tarsil commented 8 months ago

@nickdepinet i have the surface laptop 3. I had Ubuntu before but something happened and restarted and since then not only it doesn't boot from a USB (although I found a way to workaround that) but I literally did a brand new installation and wiped out the old and still doesn't boot. It remains stuck in the windows logo.

The Ubuntu was added to the UEFI but it doesn't boot at all.

Also, the the UEFI is with security boot off as well.

I didn't even get the chance to install anything else

nickdepinet commented 8 months ago

I have seen similar behavior but only when attempting to launch mokmanagement to enroll moks after secureboot configuration - your issue seems different

tarsil commented 8 months ago

Yap, there was an installation that requires that after reboot. When rebooted it was when all show started. So I wiped the whole system out. This way I could just do it from the scratch. Nothing happened. I have an OS installed that I can't access basically. I assume that after wiping out the system and install a new one, the mok wouldn't be an issue.

Only if we had a tool that could be bootable that allows to wipe the system clean and start over.

It's enfuriating not being able to move forward. I lost all my work basically

tarsil commented 8 months ago

I have seen similar behavior but only when attempting to launch mokmanagement to enroll moks after secureboot configuration - your issue seems different

It would be nice to launch the mok management or install of the needed requirements via USB live bootable flash drive. That would solve all the problems. We could install everything needed and maybe the issue would go away. I can't seem to find anything

nickdepinet commented 8 months ago

I have worked around the mok issue by booting my liveusb (linux mint 21.2), which has popped mok key enrollment and then everything works normal after that. In my case i was able to eventually get the following error to show by spamming esc

Failed to open \EFI\BOOT\mmx64.efi - Not Found
Failed to load image \EFI\BOOT\mmx64.efi - Not Found
Failed to start MokManager: Not Found
Something has gone seriously wrong: import_mok_state() failed

Which led me to my workaround

tarsil commented 8 months ago

So, what did you do for the mok issue? Downloaded the mint, burned the image and just installed? Would you mind sharing your steps?

Because I don't know what should I do about the mok issue but I assume this is the whole reason for this to actually being happening.

Because I tried installing Ubuntu, Kubuntu.... Nothing poped the security stuff and I was never able to boot after the installation. Always with the security boot disabled.

If you could share detailed steps, I would really appreciate. At least I can try that one even if it doesn't change anything in the end.

nickdepinet commented 8 months ago

turn the secureboot on and try to boot the liveusb after installation

tarsil commented 8 months ago

So, download the mint, create a USB and install, turn on the security boot and try to run the live usb?

awptechnologies commented 8 months ago

I am using debian. secure boot set to microsoft and 3rd party I imported keys, added repository and updated installed everything and so far so good. I installed the linux-surface-secureboot-mok. at this point i rebooted. on boot it froze at microsoft logo. i let it sit for 30 mins and nothing happen. i shutdown cpu and changed boot order to try windows. windows worked fine. i then made a live usb and when trying to boot to it i was getting mok errors in top left of screen and it would just shut down. only way i could get into any linux boot usb was to go to shell environment and delete dmpstore variables related to mok.

The only thing i questioned is if i should of run sudo update-grub before the reboot.

Also i have bitlocker enabled on windows partition. but i didnt think that would have anything to do with it. especially since windows still booted fine.

Surface Pro 6 with dual boot

awptechnologies commented 8 months ago

also just incase it matters im using debian trixe (testing)

awptechnologies commented 8 months ago

should i have secure boot set to microsoft only or with 3rd party

awptechnologies commented 8 months ago

I just did the process again on a completely fresh install and no good. Everything installed correctly. on reboot locked up at surface logo. ESC spamming did nothing

tarsil commented 8 months ago

only way i could get into any linux boot usb was to go to shell environment and delete dmpstore variables related to mok

@awptechnologies how did you even do this and what did you delete or run to delete?

FredEckert commented 8 months ago

Hi: I see this same issue on a Surface Pro 7, Debian 12.2, Win11 dual boot.

I believe the issue is that the MOK management console that allows you to register MOKs is not opening. I am not 100% sure on this because, I did a whole bunch of stuff while trying to try to recover my ability to boot and maybe I messed up something in the UEFI non-volatile storage. I opened an issue here: https://github.com/lcp/mokutil/issues/75

@nickdepinet, Are you saying that you can secure boot the Mint Live USB and perform mok key enrollment and the mok management console opens when you reboot?

Thanks

nickdepinet commented 8 months ago

Yes - I was able to perform mok key enrollment by booting from the Mint 21.2 Live USB while the system was stuck in the mok bootloop cycle. choosing the liveusb as the boot device opened mokmanager on boot and I could enroll the key. I can then boot the system normally.

tarsil commented 8 months ago

@nickdepinet I used your solution and IT WORKS!!! :. Thank you so so so much. The Mok issues are addressed by the mint.

@FredEckert So this is what I did:

  1. Install a distro with secure boot off.
  2. Downloaded mint 21.2.
  3. Created a LiveUSB for the Mint (following their instructions).
  4. Enabled secureboot
  5. Load from USB on boot

Mint will trigger the Mokservice and from there either you engage the keys or reset them (my case).

I hope this helps. Thank you @nickdepinet ! Really

FredEckert commented 8 months ago

Mine won't boot the mint 21.2 install USB with secure boot enabled. I get a blue security violation screen with a tiny OK button in the middle. I have Microsoft and 3rd Party keys selected.

nickdepinet commented 8 months ago

Leave secure-boot off and try to boot - only turn secureboot on once you're ready to enroll mok keys after installation

tarsil commented 8 months ago

@nickdepinet yes. Correct, apologies. First install without secure boot. Then turn on with the mint.

awptechnologies commented 8 months ago

in order to delete mok keys i used arch installer. Selected efi shell and you can type dmpstore -all to list all variables dmpstore -d -all {variable name} to delete variables... Be careful dmpstore -d -all without the variable name will delete all variables... Don't sweat if you do this by accident though. Through a ridiculous amount of research i have figured out how to restore default variables. I did this because i got mad and deleted all variables. If you happen to do this you can find the default firmware for your surface and install them. You have to be in windows though.

The link for my surface pro 6 is here https://www.microsoft.com/en-us/download/details.aspx?id=57514 Im guessing you would just search for this with your model of laptop.

So with the mint option. I would disable secure boot then install linux surface? then on reboot enable microsoft and 3rd party and boot into mint live cd?

Basically asking when should i disable secure boot?

Do i install the linux-surface kernel and linux-surface mok with secure boot enabled or without it enabled?

awptechnologies commented 8 months ago

@tarsil how did you do it? It seems you have found a solution to my exact problem. I am on the 4th full reinstall just want it to work.....

tarsil commented 8 months ago

@awptechnologies i literally followed @nickdepinet advice.

  1. Install a distro with secure boot off.
  2. Downloaded mint 21.2.
  3. Created a LiveUSB for the Mint (following their instructions).
  4. Enabled secureboot
  5. Load from USB on boot.
  6. Removed all Mok keys (I wanted that)
  7. Worked like a charm.

If you follow the instructions to add the secure mok (it's there in the instructions) and you get blocked, then use the mint to boot via USB and enroll the mok keys.

Every time this might happen (and it might), basically use the mint live usb with secure boot to enroll.

Rinse repeat

awptechnologies commented 8 months ago

do we have any idea why this is? i may be asking alot but in the next 30 mins im going to be trying again. just got windows installed and shrunk the volume to allow room for debian.

So just to clarify, im going to

Shutdown power on and disable secure boot install debian go through linux-surface kernel install and enroll linux-surface mok key shudown power on and enable secure boot with 3rd party boot from mint live usb enroll mok key with password surface reboot into my debian distro

Sound right

awptechnologies commented 8 months ago

and basically anytime there is a update to mok keys or something that requires mok re enrollment i will have to boot into mint live usb first to enroll keys at blue screen then i can reboot into my actual distro?

tarsil commented 8 months ago

I don't know why. Honestly but it sounds right. I did with Kubuntu. I love Kubuntu šŸ™‚ but its the same

tarsil commented 8 months ago

Updates should not require this. I think it's one off. Only when you ijsya something that requires mok keys which is rare (like installing a different GUI).

Even that is not certain that will always happen. I will always have mint to help me from now on but I doubt you will need after all of this.

I put all my usual software back and no issues so far

awptechnologies commented 8 months ago

i am a debian lover. Reason being all of my backend is based on it. I run proxmox on my 2 dell r730s which is based on debian. I run truenas scale for nas which is based on debian. I run debian with no desktop environment (server) for my vms that handle my docker swarm. Used to be a big ubuntu user but i fell in love with the fact debian is a community project and not backed by any one company.

Thanks so much for your help i will try this and report back with my results.

awptechnologies commented 8 months ago

also is there a reason you didnt install the mok keys and removed them instead? im guessing you just dont run secure boot?

tarsil commented 8 months ago

I do run secure boot with 3rd party yes. Reason was because I was exploring and wanted to understand the how's and whys since it happened. That was the only reason why I didn't install straightaway. I know I could but then I couldn't suggest you what I did šŸ™‚

StollD commented 8 months ago

This issue seems to be pretty common in the last few weeks.

My understanding is, that it is related to a new UEFI firmware from Microsoft, that uses a new security feature called NX. Previously, the memory returned by the UEFI was always readable, writable and executable. Now it is readable and writable by default, and need to be manually made executable.

Some programs struggle with this new feature because they seem to rely on an assumption that the memory that comes from the firmware is r-w-x. Fedoras build of GRUB is a notable example, see here: https://github.com/linux-surface/linux-surface/issues/1162

@nickdepinet Out of curiosity, when you encountered this issue, where you also running Mint 21.2? I am trying to figure out what makes the Mint LiveUSB work.

Generally it would be good if you all could post the exact distro you were running when you ran into this issue. So "Ubuntu 22.04.2" or "Ubuntu 22.04 with all the newest updates". Something like that. If we know what distributions are affected, then we can advise people to not enroll the key there until this is fixed.

Also, and I can't stress this enough, you need to report this to your distro. They are the only ones who can really fix it.

awptechnologies commented 8 months ago

Im on Latest Debian Bookworm

nickdepinet commented 8 months ago

Yes, I was running mint 21.2 when the issue occurred. (this is a fresh install on a fresh surface laptop go 3 - i havent seen this on my other laptops) In my case - the surface seems to be looking for mmx64.efi at a specific place \EFI\BOOT\mmx64.efi. the liveusb seems to have mmx64 at this location(?) - I've also seen reports for other laptops where copying mmx64.efi into the efi boot partition from any liveusb makes the system work

awptechnologies commented 8 months ago

If im not mistaken enrolling the mok keys is the only way to use linux-surface kernel with secure boot. Is this correct?

nickdepinet commented 8 months ago

that is correct - you can use the kernel without secure-boot but you must enroll the mok keys for secure boot to work

StollD commented 8 months ago

The location is relative to the shim binary that is booted by the UEFI. So if the boot entry points at \EFI\BOOT\shimx64.efi, it will try \EFI\BOOT\mmx64.efi.

But if the binary wasn't there it should throw an error like the one you posted above (https://github.com/linux-surface/linux-surface/issues/1274#issuecomment-1773464162) immideately, and not hang. Or?

I can't imagine that every debian based distro sets up their shim / MokManager wrong.

If im not mistaken enrolling the mok keys is the only way to use linux-surface kernel with secure boot. Is this correct?

You need to enroll keys. Not neccessarily our keys, you could sign the kernel yourself, but the process is always the same.

awptechnologies commented 8 months ago

last time it failed it hung on boot logo and as soon as i tried to boot to Debian live is when it immediately threw the error and shutdown.

awptechnologies commented 8 months ago

has anyone had success with copying it from mint live to a debian distro \EFI\BOOT

Also is this file located on the root partition or the efi partition?

tarsil commented 8 months ago

So, @StollD i had the same issue. I installed Ubuntu, fedora, Kubuntu. I encountered issues in all of them and the USB wasn't even bootable.

How to make it bootable? Copy the grubx64.efi and name it mmx64.efi (if I'm not mistaken). If there is a file there with the same name. Remove and make a copy of grub64x and name it the same.

This makes the USB bootable.

Then after installing it was when @nickdepinet solution became handy and worked like a charm.

Mint, apparently is the only distro where the boot is not like the other distros. The first window is the Mko Management and then the rest.

I think since MS updated the firmware (not even remotely impressed they made it harder), the next distros will probably follow through mostly because Microsoft partners now with Canonical but these are just my 2 cents.

@awptechnologies my steps I didn't need to copy anything. I did literally what I described here about the grubx64.efi + the steps I mentioned in some replies ago and that is it. My OS is working like nothing happened and with the keys enrolled.

When I enrolled the keys, I got stuck again so I used my mint live usb to boot and enroll the keys (thank you again, @nickdepinet ) and then restarted and everything was working like a charm.

tarsil commented 8 months ago

has anyone had success with copying it from mint live to a debian distro \EFI\BOOT

Also is this file located on the root partition or the efi partition?

This could probably work really well too as apparently other people did it. I went easier and even having installed Kubuntu, I simply booted my laptop with Linux mint live USB just to enroll the keys (literally the first thing that shows you when you boot it using the distro 21.2 of Mint) and then rebooted again without my USB and went straight to my Kubuntu šŸ™‚

awptechnologies commented 8 months ago

how did you create the mint live?

awptechnologies commented 8 months ago

did you just use the latest iso?

tarsil commented 8 months ago

did you just use the latest iso?

Of mint? Yes. Just that. I just needed to enroll my keys anyway and Mint does that in the boot of live usb first thing. Perfect šŸ™‚

awptechnologies commented 8 months ago

im saying there is no dedicated live usb image.

You just used the mint iso on the website? right?

For example debian offers an actual live iso

StollD commented 8 months ago

When you enroll a MOK certificate, linux sets an EFI variable that signals the shim bootloader to launch MokManager. Once you do that, live images won't work anymore, because they usually don't include MokManager, only shim. So shim fails to launch. Thats expected and how it should be.

What is not expected is the device getting stuck on the Microsoft logo. But to my understanding, this doesnt happen on live images, or?

I think since MS updated the firmware (not even remotely impressed they made it harder)

Just for the record, this has nothing to do with preventing linux from booting or something like that. NX itself makes a ton of sense, and as far as I know the writing has been on the wall since almost a year because MS stopped signing shims without NX support. Linux simply isn't ready for it.

tarsil commented 8 months ago

im saying there is no dedicated live usb image.

You just used the mint iso on the website? right?

For example debian offers an actual live iso

I just used their image and followed their instructions

awptechnologies commented 8 months ago

can you link the instructions to make it live usb?

tarsil commented 8 months ago

When you enroll a MOK certificate, linux sets an EFI variable that signals the shim bootloader to launch MokManager. Once you do that, live images won't work anymore, because they usually don't include MokManager, only shim. So shim fails to launch. Thats expected and how it should be.

What is not expected is the device getting stuck on the Microsoft logo. But to my understanding, this doesnt happen on live images, or?

I think since MS updated the firmware (not even remotely impressed they made it harder)

Just for the record, this has nothing to do with preventing linux from booting or something like that. NX itself makes a ton of sense, and as far as I know the writing has been on the wall since almost a year because MS stopped signing shims without NX support. Linux simply isn't ready for it.

It does happen. When you burn an image into a USB bootable, it won't work. You need to delete the mmx64.efi, copy the grubx64.efi and name it mmx64.efi.

Why? Don't know but only this makes the USB bootable again. Until a fix for this comes out, this is the way it can be done. Unfortunately took me days of research and losing all my work for this

tarsil commented 8 months ago

can you link the instructions to make it live usb?

https://linuxmint-installation-guide.readthedocs.io/en/latest/burn.html

But I think Rufus with GPT should do the same but I actually did what the instructions suggested.