Updates not installed on reboot: `failed to run update on reboot`

[allanday]

Attempting to install firmware updates today via GNOME Software 45.1 on Fedora Workstation 39:

• I press the "reboot and install" button
• Reboot the machine
• No update progress is shown
• After reboot, the updates are still listed

Hardware is a Lenovo ThinkPad X1 Nano Gen 2. Output of fwupdmgr get-history is:

LENOVO 21E9S0PK1L
│
├─Embedded Controller:
│ │   Device ID:          ae1abd099407b1d95698d69b7273f7fed5c6f35c
│ │   Previous version:   0.1.18
│ │   Update State:       Failed
│ │   Update Error:       failed to run update on reboot
│ │   Last modified:      2024-01-15 12:00
│ │   GUID:               62d94019-c142-4dd9-b8da-b186efd1e192
│ │   Device Flags:       • Internal device
│ │                       • Updatable
│ │                       • System requires external power source
│ │                       • Supported on remote server
│ │                       • Needs a reboot after installation
│ │                       • Device is usable for the duration of the update
│ │ 
│ └─ThinkPad X1 Nano Gen 2 Device Update:
│       New version:      0.1.20
│       Remote ID:        lvfs
│       Release ID:       69938
│       Summary:          Lenovo ThinkPad X1 Nano Gen 2 Embedded controller Firmware
│       Licence:          Proprietary
│       Size:             1.2 MB
│       Created:          2023-10-12
│       Urgency:          High
│       Vendor:           Lenovo
│       Release Flags:    • Trusted metadata
│                         • Is upgrade
│       Description:      
│       BIOS and ECFW regular update for reference code.
│       Checksum:         248e147356a9d2a401bf62fcbd957c7d1f3c2db63ae7d29215ed1bb66d55ce0a
│     
├─Intel Management Engine:
│ │   Device ID:          97b6fe9b220c7b9e1a3a1d9f404c00d4fe77ae7e
│ │   Previous version:   0.15.1662
│ │   Update State:       Failed
│ │   Update Error:       failed to run update on reboot
│ │   Last modified:      2024-01-15 12:00
│ │   GUID:               86678f88-bed8-40bb-829e-3cc7b195d4eb
│ │   Device Flags:       • Internal device
│ │                       • Updatable
│ │                       • System requires external power source
│ │                       • Supported on remote server
│ │                       • Needs a reboot after installation
│ │                       • Device is usable for the duration of the update
│ │ 
│ └─ThinkPad X1 Nano Gen 2 Corporate ME Update:
│       New version:      0.27.2176
│       Remote ID:        lvfs
│       Release ID:       49148
│       Summary:          Lenovo ThinkPad X1 Nano Gen 2 Corporate ME Firmware
│       Licence:          Proprietary
│       Size:             9.8 MB
│       Created:          2023-07-07
│       Urgency:          Medium
│       Details:          https://pcsupport.lenovo.com/de/en/search?query=N3IRM17W
│       Vendor:           Lenovo
│       Release Flags:    • Trusted metadata
│                         • Is upgrade
│       Description:      
│       Intel MEFW formal release for IPU on ADL platform project.
│       Checksum:         9b6f56c7d761d1b8cd2ae7ea441335834e4a1d9c4f0d74de36aba0bbabc1dfa5
│     
└─System Firmware:
  │   Device ID:          632acf4927c0b5fb53519d6beed3b60adb73f1d5
  │   Previous version:   0.1.14
  │   Update State:       Failed
  │   Update Error:       failed to run update on reboot
  │   Last modified:      2024-01-15 12:00
  │   GUID:               562a0522-d1a0-4b88-ad34-3cfda8788412
  │   Device Flags:       • Internal device
  │                       • Updatable
  │                       • System requires external power source
  │                       • Supported on remote server
  │                       • Needs a reboot after installation
  │                       • Cryptographic hash verification is available
  │                       • Device is usable for the duration of the update
  │ 
  └─ThinkPad X1 Nano Gen 2 System Update:
        New version:      0.1.23
        Remote ID:        lvfs
        Release ID:       69937
        Summary:          Lenovo ThinkPad X1 Nano Gen 2 System Firmware
        Licence:          Proprietary
        Size:             22.2 MB
        Created:          2023-10-11
        Urgency:          High
        Vendor:           Lenovo
        Release Flags:    • Trusted metadata
                          • Is upgrade
        Description:      
        BIOS and ECFW firmware Update For Reference code N3AET77W.
        Checksum:         edd6e1c832bd486fb2ca1e32ebadbf1ba9c2ec96966ad6bffd28532863a1ff3f

[hughsie]

@allanday do you get any message when you reboot to apply the update? you might need to use a camera phone. You'll see fwupd-efi 1.4.... then hopefully something else. Also, does it work if you turn off SecureBoot first?

[allanday]

do you get any message when you reboot to apply the update? you might need to use a camera phone.

I've taken lots of pictures of the on screen boot log. I don't see any mention of fwupd. I'd be happy to pass the photos along.

does it work if you turn off SecureBoot first?

I get the same behaviour when SecureBoot is off.

[carnage-mode]

This is a bug I've had for more than two years. I initially thought it was a shim issue that should have been fixed by a version bump and reported it (here for reference). That didn't work and I ended up opening a discussion. I've been running possibly vulnerable firmware for quite a while and I can't find any solution.

[kmauleon]

@allanday have you tried updating one firmware at a time?

[kmauleon]

Please try to reinstall MEFW. download here

"fwupdmgr --allow-older --allow-reinstall install [cab file]"

[allanday]

Please try to reinstall MEFW. download here

"fwupdmgr --allow-older --allow-reinstall install [cab file]"

I did this. After restarting gnome-software continued to list two firmware updates: ThinkPad X1 Nano Gen 2 Device Update and ThinkPad X1 Nano Gen 2 System Update. I've tried updating them both individually, but when I return to the updates list after restart they are still listed.

Here's the output of fwupdmgr get-history: fwupdmgr-get-history.txt

[kmauleon]

hi sorry i missed your last input @allanday . can you please share BIOS Setup Menu screen shot...thanks!

[kmauleon]

hi @allanday can you please update your system BIOS to 1.27

[kmauleon]

closing as no update.... let me know if need to reopen... thank you very much...

[allanday]

Sorry for missing the previous messages here.

I tried updating my firmware from the command line, but it doesn't work. Afterwards I see this:

Devices that were not updated correctly:
 • Embedded Controller (0.1.18 → 0.1.20)
 • Intel Management Engine (0.15.1662 → 1.32.2418)
 • System Firmware (0.1.14 → 0.1.28

Here's a picture of my setup menu:

[kmauleon]

hi @allanday can you please try to update one firmware at a time... System firmware first followed by EC firmware... last update MEFW

if manual update of system firmware and EC firmware did not work.. you may update via USB (Readme / BIOS_EC)

[carnage-mode]

This issue was fixed for me when I reinstated my silverblue setup (reference). You're using workstation but maybe that could resolve it for you @allanday?

[allanday]

hi @allanday can you please try to update one firmware at a time... System firmware first followed by EC firmware... last update MEFW

I tried installing the system firmware on its own and got the same result. Thanks for the suggestion.

[kmauleon]

@allanday please update via USB.... thank you very much

[kmauleon]

@allanday have you tried updating BIOS via USB? Please advise. Thank you very much.