TGL-U: Can't suspend with USB-C device connected

alexgallotta commented 3 years ago

I have a lemp10 lemur pro, I update the firmware few days ago. After the reboot and firmware installation, every time I put the laptop in sleep mode while it is connected to a USB-C hub (plugged with a monitor and mouse+keyboard) the fan starts spinning continuously after 10 seconds or so. When I resume and login, the fan stops. I tried to run fwudp for update history, but got nothing

fwupdmgr get-history                                                                             1 ↵
No history

hansmerkl commented 2 years ago

It is the firmware. But I wasn't able to determine what was causing it.

What I saw was C10_GATE# constantly asserted/deasserted, which results in:

Linux reporting some PC10 residency time

Power LED stays green because it's less that 1s

Presumably, CPU constantly being put in/out of C10 causes a temperature spike, which turns on the fan

If you want EC logs during suspend, you will need a Mega 2560 and set up EC debugging.

Do you work for System76? Sounds like this kind of issue needs a lab with proper equipment.

curiousercreative commented 2 years ago

@crawfxrd I'm new to all this, but I've got a spare rock64 that I'd be keen to tinker with. I don't see that or similar SBCs listed in your linked documentation, but do you think it's a worthwhile exploration or a total waste of time and won't work for this purpose? Looks to have i2c pins similar to the documented Adafruit Trinket M0.

Also, the i2c documentation doesn't list galp5. Should it be the same as galp4?

crawfxrd commented 2 years ago

I highly recommend you do not attempt the I2C connection over the battery. You have one chance; If you get something wrong, your board will be dead. (But yes, schematics show SMC_BAT is pin 4 and SMD_BAT is pin 5.)

crawfxrd commented 2 years ago

CB:59024

hansmerkl commented 2 years ago

CB:59024

@crawfxrd I am not an expert on coreboot so I wonder if this is related to the issue of this thread?

curiousercreative commented 2 years ago

Just received a launch keyboard (USB-C device) today and this bug reproduces with just the keyboard plugged in. I'm now on 21.04 with kernel 5.13.0-7614.

hansmerkl commented 2 years ago

Just received a launch keyboard (USB-C device) today and this bug reproduces with just the keyboard plugged in. I'm now on 21.04 with kernel 5.13.0-7614.

I am still curious what changed from 5.8 kernel. It worked fine back then.

nkalkhof commented 2 years ago

I've cherry picked CB:59024 along with CB:58798 and build coreboot based on current system76-branch for a lemp10. Suspend now works (cpu enters pc10) after a lenovo thunderbolt 3 dock is connected. After disconnecting the laptop is stuck at pc8 again! Reconnecting the dock doesn't bring s0ix back. This behaviour is consistant strating from kernel 5.12.

redoxeon commented 2 years ago

@nkalkhof

I've cherry picked CB:59024 along with CB:58798 and build coreboot based on current system76-branch for a lemp10. Suspend now works (cpu enters pc10) after a lenovo thunderbolt 3 dock is connected. After disconnecting the laptop is stuck at pc8 again! Reconnecting the dock doesn't bring s0ix back. This behaviour is consistant strating from kernel 5.12.

I think this is what I'm experiencing, but I left the comment on the wrong issue. here's the comment

Haven't connected my eGPU since I did this, but haven't been able to suspend since.

hansmerkl commented 2 years ago

Any news on this? Is there hope that this will get fixed one day?

cement-head commented 2 years ago

Yeah, okay same issue here on a galp5 with a fresh install of Pop_OS 22.04 5.17 kernel. Must be related to the firmware issue(s). Apparently on the other issue, there's a firmware update.

This (I believe) is a "wake-up" issue, not necessarily a suspend issue. In other words, the USB subsystem isn't being properly unloaded during the suspend cycle and therefore immediately resumes - giving the impression that suspend doesn't work.

hansmerkl commented 2 years ago

Seems with some of the latest updates the fan doesn't turn on anymore but the laptop still does't suspend properly. The laptop still draws around 0.2A when it goes to sleep.

epistax commented 2 years ago

I'm seeing this issue on a Lemp10 running 22.04 / 2021-07-20_93c2809. In my case, my USB-C device is a PD charger. I see the issue only in conjunction with a bluetooth speaker. I posted my details here: reddit

curiousercreative commented 2 years ago

@epistax that's a separate issue unrelated to firmware I believe. It's a 5.17 kernel bug with certain Bluetooth controllers. You can work around it by turning BT off prior to suspend or booting into old kernel. Should be fixed soon

thomas-zimmerman commented 2 years ago

I'm seeing this issue on a Lemp10 running 22.04 / 2021-07-20_93c2809. In my case, my USB-C device is a PD charger. I see the issue only in conjunction with a bluetooth speaker. I posted my details here: reddit

There are a couple fixes pending that will land in a kernel update along with a new bluez release. In testing, active bluetooth connections will still block suspend unless additional configuration changes are made

TobiasKaiser commented 2 years ago

I am experiencing the same problem on a lemp10 with a 5.10.0 kernel and a 2022-04-26_650b984 firmware image: When connected to a USB C dock (03f0:046b HP, Inc HP USB-C Dock G5), the fan turns on and stays on after entering suspend mode. This is quite annoying for me. Strange to see that there is no fix given that the problem is known for over a year now.

alexrelis commented 2 years ago

System76 firmware team, is there an update on this issue? I BEG of you to patch this bug as it severely impacts my work as I need to use a docking station for multi-monitors, keyboard/mouse, microphone, and webcam. I don't know if these laptops should be advertised as Thunderbolt capable in its current state.

hansmerkl commented 2 years ago

System76 firmware team, is there an update on this issue? I BEG of you to patch this bug as it severely impacts my work as I need to use a docking station for multi-monitors, keyboard/mouse, microphone, and webcam. I don't know if these laptops should be advertised as Thunderbolt capable in its current state.

Totally agree. This situation is ridiculous. It's pretty bad that such obvious flaws aren't being addressed.

cement-head commented 2 years ago

Kernel 5.18 fixes a lot of issues...

curiousercreative commented 2 years ago

Kernel 5.18 fixes a lot of issues...

This is not a kernel issue, it's firmware.

hansmerkl commented 2 years ago

Kernel 5.18 fixes a lot of issues...

It was fine with kernel 5.8 but then the problems started. 5.18 definitely doesn't fix the problem and I believe it's firmware.

nkalkhof commented 2 years ago

Kernel 5.18 fixes a lot of issues...

This is not a kernel issue, it's firmware.

I totally agree. After more than one year the issue still has not being properly addressed. Support Tickets remain unanswered. Thunderbolt 4 for example does not work at all unless you use a patched kernel from pop!os. Very disappointing for devices in that price range!

curiousercreative commented 2 years ago

Thunderbolt 4 for example does not work at all unless you use a patched kernel from pop!os

Are you saying that Thunderbolt 4 doesn't work on other distros with these S76 laptops or are you saying that you are running a patched kernel on Pop!_OS to make TB4 work? TB4 works on my galp5 on Pop! 22.04 aside from this bug.

hansmerkl commented 2 years ago

Kernel 5.18 fixes a lot of issues...

This is not a kernel issue, it's firmware.

I totally agree. After more than one year the issue still has not being properly addressed. Support Tickets remain unanswered. Thunderbolt 4 for example does not work at all unless you use a patched kernel from pop!os. Very disappointing for devices in that price range!

What patched kernel? TB seems to be working for me except the suspend/fan issues.

alexrelis commented 2 years ago

If you can't suspend your laptop, experience the fan constantly going on and off and one of your CPU cores spike to 100% every time you plug in a Thunderbolt accessory then I don't think you could call it "working".

hansmerkl commented 2 years ago

If you can't suspend your laptop, experience the fan constantly going on and off and one of your CPU cores spike to 100% every time you plug in a Thunderbolt accessory then I don't think you could call it "working".

something "works" but certainly not "works properly"

alexrelis commented 2 years ago

I see what you mean, but I guess my point is this: if using a Thunderbolt accessory brings about so many problems that you avoid using Thunderbolt accessories altogether, then it's as good as the computer not having Thunderbolt support at all.

hansmerkl commented 2 years ago

I see what you mean, but I guess my point is this: if using a Thunderbolt accessory brings about so many problems that you avoid using Thunderbolt accessories altogether, then it's as good as the computer not having Thunderbolt support at all.

Yeah. This definitely makes me reconsider buying System76. The systems don't seem to be fine tuned for Linux as advertised.

jacobgkau commented 2 years ago

Thank you everyone for making your opinions heard. This issue is known and that is why the issue tracker page is here and marked as "Open." Discussion solely about dissatisfaction is not helpful to our firmware developers and does not contribute to development.

I'm still hoping our firmware developers can come up with a solution to this eventually; from Tim linking https://review.coreboot.org/c/coreboot/+/59024 earlier, it sounds like we may switch these systems from S0ix suspend to S3 suspend in a future firmware update, which may fix this.

crawfxrd commented 2 years ago

When I'm not buried in ADL boards, I'll look into this again along with the lemp10 SSD issues.

leviport commented 2 years ago

https://github.com/system76/ec/pull/295 may resolve this issue, so I am unlocking the comments here in case anyone wants to test that PR and provide feedback. Let's keep the discussion focused on the technical details of this bug and the proposed solution.

hansmerkl commented 2 years ago

system76/ec#295 may resolve this issue, so I am unlocking the comments here in case anyone wants to test that PR and provide feedback. Let's keep the discussion focused on the technical details of this bug and the proposed solution.

Is there a way to test this on a galp5?

crawfxrd commented 2 years ago

See: flashing.md#internal-programmer

curiousercreative commented 2 years ago

@leviport @crawfxrd I've only just flashed this so I'll report any regressions I encounter over time, but a quick test shows positive results!

With galp5, connect a launch_1 keyboard to the TB4 port with a TB3/4 cable. ER/AR: galp5 enters and exits suspend as expected, blinking LED, no fans

I don't have access right now to the the USB-C monitor that I first noticed this behavior on, but I'll likely report results there sometime this month.

TobiasKaiser commented 2 years ago

I have just flashed the EC firmware from https://github.com/system76/ec/pull/295 on my lemp10. The suspend/fan-on bug is not occuring anymore for me anymore, which is great. From my observations so far, the update did not introduce any new bugs.

trevor87 commented 2 years ago

Great to hear that this is fixed! Is there already a rough estimate when this fix will be available via the normal firmware update?

alexrelis commented 2 years ago

Great work guys. Let us know when the firmware update gets pushed out to users.

crawfxrd commented 2 years ago

lemp10 can suspend with a USB device plugged in, but it seems TBT devices still have a problem resuming. Testing with an eGPU, the bridge (and so the eGPU) fails to go from D3 to D0.

I'm not too concerned about the eGPU, but it was also reported for an external enclosure if the drive was mounted when suspending.

kernel: pcieport 0000:02:00.0: Unable to change power state from D3cold to D0, device inaccessible
kernel: pcieport 0000:03:01.0: Unable to change power state from D3cold to D0, device inaccessible
kernel: pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible
kernel: pci 0000:04:00.3: Unable to change power state from D3cold to D0, device inaccessible
kernel: xhci_hcd 0000:04:00.2: Unable to change power state from D3cold to D0, device inaccessible
kernel: snd_hda_intel 0000:04:00.1: Unable to change power state from D3cold to D0, device inaccessible

trevor87 commented 2 years ago

Just flashed the current ec @ 7f28764 and my standby problems with my usb-c monitor/hub seem to have gone. Thanks!

jacobgkau commented 1 year ago

oryp8's latest released firmware is able to suspend/resume with a USB-C device connected, but fails to resume with a Thunderbolt device connected (same as what Tim described for lemp10 on development firmware.)

crawfxrd commented 1 year ago

2022-11-21_b337ac6 has been released for all TGL-U models.

system76 / firmware-open

TGL-U: Can't suspend with USB-C device connected #199

Yeah, okay same issue here on a galp5 with a fresh install of Pop_OS 22.04 5.17 kernel. Must be related to the firmware issue(s). Apparently on the other issue, there's a firmware update.