search

dell / thunderbolt-nvm-linux

Thunderbolt NVM updates for Linux
104 stars 12 forks source link

Thunderbolt firmwarm fails after Thunderbolt NVM (33) XPS 9370 #17

Closed steevee closed 5 years ago

steevee commented 6 years ago

Thunderbolt Controller has disappeared post-upgrade. Firmware may be corrupted(?) Can you advise how to manually reinstall the driver?

OS: Ubuntu 18.04 (and 18.10) Kernel: 4.15 (and 4.18) Product Name: XPS 13 9370 BIOS Version: 1.5.1 Thunderbolt BIOS security level: none

Steps: Thunderbolt NVM (33) was installed via Ubuntu Software Installer (after prompting for update) with a secondary display active (not sleeping). Post successfully installing, the prompt reappeared several times (similarly to #8 but consecutively). Post reboot could not wake thunderbolt ports (still takes power, but nothing else).

dmesg

Thunderbolt firmware fails to start.

...
[    7.775524] thunderbolt 0000:05:00.0: enabling device (0000 -> 0002)
[    7.776832] intel-lpss 0000:00:15.0: enabling device (0000 -> 0002)
[    7.777102] i801_smbus 0000:00:1f.4: SPD Write Disable is set
[    7.777167] i801_smbus 0000:00:1f.4: SMBus using PCI interrupt
[    7.777419] thunderbolt 0000:05:00.0: NHI initialized, starting thunderbolt
[    7.777422] thunderbolt 0000:05:00.0: allocating TX ring 0 of size 10
[    7.777441] thunderbolt 0000:05:00.0: allocating RX ring 0 of size 10
[    7.777456] thunderbolt 0000:05:00.0: control channel created
[    7.777456] thunderbolt 0000:05:00.0: control channel starting...
[    7.777458] thunderbolt 0000:05:00.0: starting TX ring 0
[    7.777473] thunderbolt 0000:05:00.0: enabling interrupt at register 0x38200 bit 0 (0x0 -> 0x1)
[    7.777474] thunderbolt 0000:05:00.0: starting RX ring 0
[    7.777489] thunderbolt 0000:05:00.0: enabling interrupt at register 0x38200 bit 12 (0x1 -> 0x1001)
[    7.777502] thunderbolt 0000:05:00.0: starting ICM firmware
[    7.777503] thunderbolt 0000:05:00.0: could not start ICM firmware # NOTE: error
[    7.777530] thunderbolt 0000:05:00.0: stopping RX ring 0
[    7.777541] thunderbolt 0000:05:00.0: disabling interrupt at register 0x38200 bit 12 (0x1001 -> 0x1)
[    7.777624] thunderbolt 0000:05:00.0: stopping TX ring 0
[    7.777635] thunderbolt 0000:05:00.0: disabling interrupt at register 0x38200 bit 0 (0x1 -> 0x0)
[    7.777640] thunderbolt 0000:05:00.0: control channel stopped
[    7.777650] thunderbolt 0000:05:00.0: freeing RX ring 0
[    7.777665] thunderbolt 0000:05:00.0: freeing TX ring 0
[    7.777676] thunderbolt 0000:05:00.0: shutdown
...

lspci -nn

Thunderbolt device appears to exist (whether devices connected or not):

...
05:00.0 System peripheral [0880]: Intel Corporation DSL6540 Thunderbolt 3 NHI [Alpine Ridge 4C 2015] [8086:1577] (rev 02)
...

lsusb

Does not list any devices connected to thunderbolt ports (nor do any devices function)

lsmod | grep thunderbolt

Thunderbolt kernel modules present:

intel_wmi_thunderbolt    16384  0
thunderbolt           118784  0
wmi                    24576  5 intel_wmi_thunderbolt,dell_wmi,wmi_bmof,dell_smbios,dell_wmi_descriptor

fwupdmgr get-devices --show-all-devices

No longer lists XPS 9370 Thunderbolt Controller as a device (whether devices connected or not):

Intel AMT [unprovisioned]
  DeviceId:             e2623122c99d58220498aacbfcfdb1baebbae3c5
  ParentDeviceId:       8a21cacfb0a8d2b30c5ee9290eb71db021619f8b
  Guid:                 2800f812-b7b4-2d4b-aca8-46e0ff65814c
  Summary:              Hardware and firmware technology for remote out-of-band management
  Plugin:               amt
  Flags:                internal|registered
  Vendor:               Intel Corporation
  Version:              11.8.55
  VersionBootloader:    11.8.55
  Icon:                 computer
  Created:              2018-11-28

XPS 13 9370 System Firmware
  DeviceId:             8a21cacfb0a8d2b30c5ee9290eb71db021619f8b
  Guid:                 7ceaf7a8-0611-4480-9e30-64d8de420c7c
  Guid:                 43ea5588-d9a4-5031-8ad3-308045302d6b
  Guid:                 230c8b18-8d9b-53ec-838b-6cfc0383493a
  Plugin:               uefi
  Flags:                internal|updatable|require-ac|supported|registered|needs-reboot
  Version:              0.1.5.1
  VersionLowest:        0.1.5.1
  Icon:                 computer
  Created:              2018-11-28

KXG50ZNV512G NVMe TOSHIBA 512GB
  DeviceId:             f954c7acdf5fab61aeaca1cd71d29ea5ade6992f
  Guid:                 4d0aed03-a30c-52c6-99e7-a8977797c3d9
  Guid:                 ad9fe8f7-cdc4-52c9-9fea-31b6f4988ffa
  Serial:               Z7BS10IYTY7T
  Summary:              NVM Express Solid State Drive
  Plugin:               nvme
  Flags:                internal|updatable|require-ac|registered|needs-reboot
  VendorId:             NVME:0x1179
  Version:              AADA4102
  Icon:                 drive-harddisk
  Created:              2018-11-28

Unifying Receiver
  DeviceId:             66a1e4c324ce716554ac74b67bf0a6e13ea6f583
  Guid:                 279ed287-3607-549e-bacc-f873bb9838c4
  Guid:                 21e75d9a-5ce6-5da2-b7ab-910c7f3f6836
  Guid:                 9d131a0c-a606-580f-8eda-80587250b8d6
  Summary:              A miniaturised USB wireless receiver
  Plugin:               unifying
  Flags:                supported|registered
  Vendor:               Logitech
  VendorId:             USB:0x046D
  Version:              RQR12.07_B0029
  VersionBootloader:    BOT01.02_B0014
  Icon:                 preferences-desktop-keyboard
  Created:              2018-11-28

fwupdmgr install

Cannot reinstall manually downloaded cab file:

$ fwupdmgr install ~/Downloads/925aaf439fc1b66aea20fb3868534e3853902c9d-ThunderboltFirmwareUpdateLinux_4.33.18.004.cab -v
(fwupdmgr:15249): Fwupd-DEBUG: Emitting ::status-changed() [decompressing]
Decompressing?         [-                                      ](fwupdmgr:15249): Fwupd-DEBUG: Emitting ::status-changed() [idle]
Decompressing?         [***************************************]
No supported devices found

Having some serious downtime trying to resolve. Any help much appreciated!

andrejpodzimek commented 5 years ago

I have fwupd 1.2.1. I don't use tbt. Here's a "dump" from boltctl when the SSD is not connected:

# boltctl list
 ○ SAMSUNG ELECTRONICS CO., LTD Portable SSD X5
   ├─ type:          peripheral
   ├─ name:          Portable SSD X5
   ├─ vendor:        SAMSUNG ELECTRONICS CO., LTD
   ├─ uuid:          00cc58e5-dcb7-7501-ffff-ffffffffffff
   ├─ status:        disconnected
   ├─ authorized:    So 8. prosince 2018, 07:27:10 UTC
   ├─ connected:     So 8. prosince 2018, 07:27:06 UTC
   └─ stored:        So 8. prosince 2018, 05:31:09 UTC
      ├─ policy:     auto
      └─ key:        yes

And here's a connected case:

# boltctl list
 ● SAMSUNG ELECTRONICS CO., LTD Portable SSD X5
   ├─ type:          peripheral
   ├─ name:          Portable SSD X5
   ├─ vendor:        SAMSUNG ELECTRONICS CO., LTD
   ├─ uuid:          00cc58e5-dcb7-7501-ffff-ffffffffffff
   ├─ status:        authorized
   │  ├─ domain:     cd010000-0080-7f08-a371-43128a038a21
   │  └─ authflags:  secure
   ├─ authorized:    So 8. prosince 2018, 08:04:41 UTC
   ├─ connected:     So 8. prosince 2018, 08:04:37 UTC
   └─ stored:        So 8. prosince 2018, 05:31:09 UTC
      ├─ policy:     auto
      └─ key:        yes

When I read from the SSD, speeds are around 2.3 GB/s (18.4 Gb/s), which is quite impressive. Here's what fwupdmgr says:

# fwupdmgr get-devices
XPS 13 9370 Thunderbolt Controller
  DeviceId:             91cb4290ca585a69eb6c7155ca9f84ac633cb6f0
  Guid:                 4eeb9d07-a96c-56d6-92d3-4a23ee7a6e4a
  Summary:              Unmatched performance for high-speed I/O
  Plugin:               thunderbolt
  Flags:                internal|updatable|supported|registered
  Vendor:               Dell
  VendorId:             TBT:0x00D4
  Version:              33.00
  Icon:                 computer
  Created:              2018-12-08

XPS 13 9370 System Firmware
  DeviceId:             8a21cacfb0a8d2b30c5ee9290eb71db021619f8b
  Guid:                 7ceaf7a8-0611-4480-9e30-64d8de420c7c
  Guid:                 230c8b18-8d9b-53ec-838b-6cfc0383493a
  Guid:                 43ea5588-d9a4-5031-8ad3-308045302d6b
  Plugin:               uefi
  Flags:                internal|updatable|require-ac|supported|registered|needs-reboot
  Version:              0.1.6.3
  VersionLowest:        0.1.6.3
  Icon:                 computer
  Created:              2018-12-08
  UpdateState:          success

XPS 13 9370 TPM 2.0
  DeviceId:             370e10407b1f04ade798a9f1d3e1fa57c67750c3
  Guid:                 cb8da68d-cd80-5f5b-8fef-038383adbb83
  Summary:              Platform TPM device
  Plugin:               uefi
  Flags:                internal|updatable|require-ac|registered|needs-reboot
  Vendor:               Dell Inc.
  Version:              7.2.0.1
  Icon:                 computer
  Created:              2018-12-08

KXG50ZNV1T02 NVMe TOSHIBA 1024GB
  DeviceId:             f954c7acdf5fab61aeaca1cd71d29ea5ade6992f
  Guid:                 4d0aed03-a30c-52c6-99e7-a8977797c3d9
  Guid:                 ad9fe8f7-cdc4-52c9-9fea-31b6f4988ffa
  Serial:               88RS10J9T8KQ
  Summary:              NVM Express Solid State Drive
  Plugin:               nvme
  Flags:                internal|updatable|require-ac|registered|needs-reboot
  Vendor:               Toshiba America Info Systems
  VendorId:             NVME:0x1179
  Version:              AADA4105
  Icon:                 drive-harddisk
  Created:              2018-12-08

Portable SSD X5
  DeviceId:             f99d0eff1001b4c29940f5df8dcd903c778b723e
  Guid:                 ed2454a2-0f06-5e01-9708-ff961fa07a92
  Plugin:               thunderbolt
  Flags:                updatable|registered
  Vendor:               SAMSUNG ELECTRONICS CO., LTD
  VendorId:             TBT:0x0175
  Version:              35.00
  Icon:                 audio-card
  Created:              2018-12-08

I think I'm starting to understand my mistake: Before the update, I should have probably started (and left blocked) boltctl power, which is the command that keeps the Thunderbolt on even when no devices are connected. Unfortunately, I just didn't have anything connected during the update. Only now did I realize that something probably should have been connected, which is slightly counter-intuitive.

steevee commented 5 years ago

TL;DR a replacement motherboard and fresh FW update worked, albeit with an error message.

Full story: Dell support replaced the motherboard on the XPS13 9370. I waited a few days before tentatively running fwupdmgr refresh and fwupmgr update (with a display port plugged into TBT).

Downloading 33.00 for XPS 9370 Thunderbolt Controller...
Fetching firmware https://fwupd.org/downloads/925aaf439fc1b66aea20fb3868534e3853902c9d-ThunderboltFirmwareUpdateLinux_4.33.18.004.cab
Downloading?           [***************************************]
Decompressing?         [***************************************]
Authenticating?        [***************************************]
Updating XPS 9370 Thunderbolt Controller?                      ]
Restarting device?     [***************************************]failed to wait for replug after update: device c1bdcc794c3849ad7c2ca8956e91ed924fd4af84 did not come back

On the final step - Restart device? - the external monitor reconnected twice over approximately 20-30s, before the message failed to wait for replug after update: device c1bdcc794c3849ad7c2ca8956e91ed924fd4af84 did not come back was displayed.

Ominously I ran fwupdmgr get-devices and received:

XPS 13 9370 System Firmware
  DeviceId:             8a21cacfb0a8d2b30c5ee9290eb71db021619f8b
  Guid:                 7ceaf7a8-0611-4480-9e30-64d8de420c7c
  Guid:                 43ea5588-d9a4-5031-8ad3-308045302d6b
  Guid:                 230c8b18-8d9b-53ec-838b-6cfc0383493a
  Plugin:               uefi
  Flags:                internal|updatable|require-ac|supported|registered|needs-reboot
  Version:              0.1.5.1
  VersionLowest:        0.1.5.1
  Icon:                 computer
  Created:              2018-12-12
  Modified:             2018-12-12
  UpdateState:          needs-reboot

XPS 13 9370 TPM 2.0
  DeviceId:             370e10407b1f04ade798a9f1d3e1fa57c67750c3
  Guid:                 cb8da68d-cd80-5f5b-8fef-038383adbb83
  Guid:                 6a704926-cdc4-53a0-84e7-227b5f1140dc
  Summary:              Platform TPM device
  Plugin:               uefi
  Flags:                internal|updatable|require-ac|registered|needs-reboot
  Vendor:               Dell Inc.
  Version:              7.2.0.1
  Icon:                 computer
  Created:              2018-12-12

KXG50ZNV512G NVMe TOSHIBA 512GB
  DeviceId:             f954c7acdf5fab61aeaca1cd71d29ea5ade6992f
  Guid:                 4d0aed03-a30c-52c6-99e7-a8977797c3d9
  Guid:                 ad9fe8f7-cdc4-52c9-9fea-31b6f4988ffa
  Summary:              NVM Express Solid State Drive
  Plugin:               nvme
  Flags:                internal|updatable|require-ac|registered|needs-reboot
  VendorId:             NVME:0x1179
  Version:              AADA4102
  Icon:                 drive-harddisk
  Created:              2018-12-12

________________________________________________

Devices that were not updated correctly:

 ? XPS 9370 Thunderbolt Controller (23.00 ? 33.00)

Uploading firmware reports helps hardware vendors to quickly identify failing and successful updates on real devices.
Upload report now? (Requires internet connection) [Y|n]: Y
Target:                  https://fwupd.org/lvfs/firmware/report
Payload:                 {
                           "ReportVersion" : 2,
                           "MachineId" : "509eb36a99b62ea474790fbe7193ed63e90813acac4252744c7a30bd4dabfe4b",
                           "Metadata" : {
                             "DistroId" : "ubuntu",
                             "DistroVersion" : "18.10"
                           },
                           "Reports" : [
                             {
                               "Checksum" : "93e5bdf16c599ae99b80084a07b0e36d8e39f843",
                               "UpdateState" : 3,
                               "Guid" : "4eeb9d07-a96c-56d6-92d3-4a23ee7a6e4a",
                               "Plugin" : "thunderbolt",
                               "VersionOld" : "23.00",
                               "VersionNew" : "33.00",
                               "Flags" : 35,
                               "Created" : 1544614940,
                               "Modified" : 1544615033,
                               "Metadata" : {
                                 "CpuArchitecture" : "x86_64",
                                 "RuntimeVersion(org.freedesktop.fwupd)" : "1.1.4",
                                 "CompileVersion(org.freedesktop.fwupd)" : "1.1.4",
                                 "CompileVersion(com.redhat.fwupdate)" : "12",
                                 "ThunderboltNative" : "False",
                                 "CompileVersion(org.freedesktop.gusb)" : "0.3.0",
                                 "CompileVersion(org.freedesktop.appstream-glib)" : "0.7.9",
                                 "RuntimeVersion(org.freedesktop.appstream-glib)" : "0.7.9",
                                 "KernelVersion" : "4.18.20-041820-generic",
                                 "CompileVersion(com.redhat.efivar)" : "35",
                                 "RuntimeVersion(com.dell.libsmbios)" : "2.4",
                                 "BootTime" : "1544614922",
                                 "RuntimeVersion(com.redhat.fwupdate)" : "12"
                               }
                             }
                           ]
                         }
Proceed with upload? [Y|n]:

Which I duly proceeded to upload.

I tried to run fwupdmgr update ---allow-reinstall but with a pending reboot required, I opted to reboot.

Thankfully post FW updates, no errors in dmesg, display port is connected via TBT fine (as it wasn't in my initial issue), and TBT controller listed as v33.0.

XPS 13 9370 Thunderbolt Controller
  DeviceId:             c1bdcc794c3849ad7c2ca8956e91ed924fd4af84
  Guid:                 4eeb9d07-a96c-56d6-92d3-4a23ee7a6e4a
  Summary:              Unmatched performance for high-speed I/O
  Plugin:               thunderbolt
  Flags:                internal|updatable|supported|registered
  Vendor:               Dell
  VendorId:             TBT:0x00D4
  Version:              33.00
  Icon:                 computer
  Created:              2018-12-12

Yay! :disappointed_relieved:

superm1 commented 5 years ago

Well glad the HW replacement sorted out the problem. This other behavior you encountered in upgrading the new board I wouldn't say is "correct".

My expectation from the above description is that the TBT controller was actually asleep when you first invoked the upgrade. Fwupd used either the kernel directly or it used bolt to power the controller up.

Can you open up a separate bug at github.com/hughsie/fwupd to diagnose it specifically?

Things that would be relevant:

superm1 commented 5 years ago

Oh wait I see in your above report - fwupd 1.1.4, kernel 4.18 and Ubuntu 18.10. So that's bolt 0.5 is in the mix.

baybal commented 5 years ago

Can we reopen this? I ran into the same issue with 9360 0x82A after flashing latest NVM.

lspci not showing the thunderbolt controller, forcing power does not work

baybal commented 5 years ago

I managed to make it to show in lspci after cold boot after 4 second shutdown button press. But it is still unresponsive even after forcing power to on.

dmesg |grep -i tbt
[    0.003785] ACPI: SSDT 0x0000000064F9B6D0 0000DB (v02 INTEL  TbtTypeC 00000000 INTL 20160422)
dmesg |grep -i thunder
(nothing)
lspci 
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 08)
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 620 (rev 07)
00:04.0 Signal processing controller: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem (rev 08)
00:14.0 USB controller: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller (rev 21)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-LP Thermal subsystem (rev 21)
00:15.0 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #0 (rev 21)
00:15.1 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #1 (rev 21)
00:16.0 Communication controller: Intel Corporation Sunrise Point-LP CSME HECI #1 (rev 21)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #1 (rev f1)
00:1c.4 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #5 (rev f1)
00:1c.5 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Intel(R) 100 Series Chipset Family LPC Controller/eSPI Controller - 9D4E (rev 21)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-LP PMC (rev 21)
00:1f.3 Audio device: Intel Corporation Sunrise Point-LP HD Audio (rev 21)
00:1f.4 SMBus: Intel Corporation Sunrise Point-LP SMBus (rev 21)
01:00.0 PCI bridge: Intel Corporation DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]
02:00.0 PCI bridge: Intel Corporation DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]
02:01.0 PCI bridge: Intel Corporation DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]
02:02.0 PCI bridge: Intel Corporation DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]
39:00.0 USB controller: Intel Corporation DSL6340 USB 3.1 Controller [Alpine Ridge]
3a:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
3b:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS525A PCI Express Card Reader (rev 01)
3c:00.0 Non-Volatile memory controller: Toshiba America Info Systems Device 0116
baybal@xps ~ $ sudo fwupdmgr --allow-reinstall --allow-older --show-all-devices get-devices
XPS 13 9360 System Firmware
  DeviceId:             3f0ecc7823ee99c718693e8ecca38f54e8738dc5
  Guid:                 5ffdbc0d-f340-441c-a803-8439c8c0ae10
  Plugin:               uefi
  Flags:                internal|require-ac|supported|registered|needs-reboot
  Version:              0.2.10.0
  VersionLowest:        0.2.10.0
  Icon:                 computer
  Created:              2019-03-15
  UpdateError:          Unable to determine EFI system partition location, override using OverrideESPMountPoint in uefi.conf

UHD Graphics 620
  DeviceId:             bbbf1ce3d1cf15550c3760b354592040292415bb
  Guid:                 fed2efa4-7045-55a1-b4fc-b29283d59fe5 <- PCI\VEN_8086&DEV_5917&REV_07
  Guid:                 8b72a10c-1279-5f8e-a28a-34fb11a58240 <- PCI\VEN_8086&DEV_5917
  Guid:                 afb8c4ec-b9a5-5cd3-aaef-cc976ce37eb8 <- PCI\VEN_8086
  Plugin:               udev
  Flags:                internal|registered
  Vendor:               Intel Corporation
  VendorId:             PCI:0x8086
  Version:              07
  Icon:                 audio-card
  Created:              2019-03-15

KXG50ZNV256G NVMe TOSHIBA 256GB
  DeviceId:             e11623b2caa18fee292058a5c09ca4e6152f7ecf
  Guid:                 4d0aed03-a30c-52c6-99e7-a8977797c3d9 <- NVME\VEN_1179&DEV_0116
  Guid:                 ad9fe8f7-cdc4-52c9-9fea-31b6f4988ffa <- NVME\VEN_1179
  Guid:                 e67bb8f0-4e33-53f7-9f43-249a203b67fb <- KXG50ZNV256G NVMe TOSHIBA 256GB
  Serial:               18BF206VFQ9S
  Summary:              NVM Express Solid State Drive
  Plugin:               nvme
  Flags:                internal|updatable|require-ac|registered|needs-reboot
  Vendor:               Toshiba America Info Systems
  VendorId:             NVME:0x1179
  Version:              AADA4106
  Icon:                 drive-harddisk
  Created:              2019-03-15
superm1 commented 5 years ago

FYI --allow-reinstall --allow-older does nothing with get-devices

I would recommend to please open a separate issue to not confused these two. So please open a new issue and do the following: