Planck EZ freezes when any key is pressed while suspended

vectorstorm commented 5 years ago

Describe the Bug

If the Planck EZ is receiving power but is suspended (for example, if the computer it's connected to is cleanly powered off but its USB ports are still providing power), and you press either the 'raise' or 'lower' key, the layer light for the raise or lower light comes on and stays on, even if the key is then released. Any RGB matrix animation freezes. If you now turn the computer back on or otherwise un-suspend the keyboard, you'll find the keyboard doesn't resume and no keypresses are registered.

This only seems to happen while the keyboard is in that suspended state.

Unplugging the keyboard and plugging it back in again causes it to return to normal functionality.

System Information

Keyboard: Planck EZ
Operating system: Tested with Ubuntu 19.04, but only happens when not actively connected to the OS.
AVR GCC version: 5.4.0
ARM GCC version: 7.3.1
QMK Firmware version: 0.6.404 (current master at time of posting this issue). Also happens with the default firmware file available from here
Any keyboard related software installed? no

Additional Context

I've tested this in a bunch of different ways. Here are some things which haven't helped:

Entirely disabling calls to palClearPad and palSetPad during layer_state_set_kb() and layer_state_set_user(). With this, the layer LEDs don't come on when you press 'raise' or 'lower', but the keyboard still freezes if you press them while the keyboard is suspended.

Adding #define RGB_DISABLE_WHEN_USB_SUSPENDED true. With this, keyswitch LEDs turn off when suspend happens, which makes it a lot easier to tell when the keyboard is actually suspended, but otherwise no change to the freeze.

Adding NO_SUSPEND_POWER_DOWN=yes to the keymap's rules.mk. Seems to make no difference.

I'd be quite happy to do more testing, if folks have ideas for other things to try!

vectorstorm commented 5 years ago

Further testing has shown that pressing any key causes the freeze; I'd just noticed the 'raise' and 'lower' keys first because of their visible layer lights.

I have consistent repros for this on two different computers, each running both Windows and Ubuntu. Can't repro it on a 2012 MacBook Pro, as the MBP kills power to its USB ports when it shuts down, so the keyboard simply loses power, rather than going into suspended mode. I only have one keyboard to test with, so if someone else can verify whether it affects them as well, that'd help show whether it's just my keyboard, or if it's a wider issue than that.

EDIT: Have now managed to get it to repro on MBP as well, by pressing a key in the middle of a restart. The RGB_DISABLE_WHEN_USB_SUSPENDED #define makes it much easier to spot the moment to press a key to freeze up the board.

Worth noting that it only happens if the keyboard has been connected and functional before the USB connection is suspended; plugging the keyboard into a computer which is already off and typing on it does not trigger this freeze.

drashna commented 5 years ago

Unfortunately, I cannot reproduce this, at all.

Howe exactly is the Planck EZ connected to the system?

vectorstorm commented 5 years ago

Thanks for attempting a repro!

The Planck EZ is generally connected via a USB C->A cable (I've used three different ones; all show the same behaviour), and I've also used the USB-C cable + C->A adapter which came with the keyboard. (I don't have a computer with a USB-C port to test with, to remove the C->A conversion from the discussion).

I've done this with the keyboard connected directly to the PC itself, with it connected chained through a monitor's USB port, and with it connected to a separate (powered) USB hub. None of these seemed to make any difference; turn the computer off, press a key on the keyboard, and the keyboard freezes until it's been removed from power and reconnected.

On the 2012 MBP, I had it plugged into the right-side USB port. There, shutting down the computer powered off the USB ports, so I had to instead do a restart. Pressing a key in the middle of that restart caused the same freeze; inoperable keyboard when the restart completed, until the keyboard was unplugged and replugged again.

vectorstorm commented 5 years ago

As a workaround, I have added this line to my rules.mk file:

NO_USB_STARTUP_CHECK = yes # Disable usb suspend check after keyboard startup

This has entirely worked around the freezing for me. I have no idea whether it ought to have done so, but whatever was going wrong for me, disabling the whole check for USB suspension appears to have sidestepped the issue. And if nobody else is able to repro the problem at all anyhow, then maybe this issue can be closed?

drashna commented 5 years ago

Okay. If that's the case, it may be an issue with the host side USB. the NO_USB_STARTUP_CHECK basically stops the firmware from waiting on the USB to be ready, and just starts everything regardless.

vectorstorm commented 5 years ago

No; you'd think that NO_USB_STARTUP_CHECK would skip the startup check, but if you check tmk_core/protocol/chibios/main.c, you'll see that it actually skips checking for the USB link going into USB_SUSPENDED during the main loop, after startup has already happened (the relevant code begins on line 188 of that file).

Skipping the "wait for USB startup" loop is actually accomplished by setting WAIT_FOR_USB to false; the code for that happens on line 148 of the file.

I haven't debugged too far into this yet, but.. based on the behaviour I'm seeing, it looks like my keyboard gets stuck inside usbWakeupHost(), on line 202 of that same file (when I haven't set the workaround that skips processing the suspended state entirely). I'll keep digging and see what I come up with!

vectorstorm commented 5 years ago

Okay, I've now reproduced this issue with two more Planck EZs (making three in total), admittedly all on those same two test computers, and have tracked the freeze down much further.

The freeze I'm seeing is happening inside usbWakeupHost() (called from around line 200 in chibios's main()); the keyboard has detected that a key has been pressed while USB communication is suspended, and so is trying to wake up the host.

Specifically, the freeze seems to be happening inside usb_lld_wakeup_host()'s call to osalThreadSleepMilliseconds(), inside the lib/chibios submodule, where the code has set a "resume" USB control flag, and is trying to wait for a few milliseconds before unsetting it again. If I comment out that sleep call, then the keyboard continues to work properly, and comes back awake correctly as soon as the computer is powered on again.

Going to keep chasing this, but.. it's starting to move into code that's a lot lower-level than I'm usually hacking on!

adrientetar commented 5 years ago

For what it's worth, I have the same issue on STM32F072. (Is this a duplicate of #5585?)

Also, I wonder if a ChibiOS upgrade might fix this as it hasn't been upgraded for a while afaict.

drashna commented 5 years ago

@adrientetar It may be related to chibiOS, but updating chibiOS isn't going to be simple. Especially not in a way that maintains backward compatibility.

vectorstorm commented 5 years ago

I agree, this sounds very very similar to what's been reported in #5585, but the mitigation proposed in that issue (adding wait_ms(5000); just before the call to send_keyboard_report();) doesn't work for me; for me, the freeze happens while still inside the inner USB Suspended loop, and it doesn't get far enough out to hit the new wait_ms(5000) line.

Also note that that issue is about waking a computer from a suspend state, while this one is about a computer which has actually been shut down. The keyboard doesn't leave the USB Suspended state at all before the freeze happens. Attempts to "resume" ought to just fail because the computer is actually switched off; it's just still providing power on the USB ports.

Daffclay commented 5 years ago

This happens on the dz60rgb aswell. If I press any key on the keyboard when the pc is off, I know for sure the keyboard won't work. Only if I replug the usb c will it function again. Anyone with the same pcb try it out?

jackhumbert commented 5 years ago

@vectorstorm can you try out this .bin (extract from the .zip) when you have the chance? It's specifically for the Planck EZ, and implements this from a couple months ago - not really sure if it will address this, but it should reset the keyboard anytime there's a freeze like this. Obviously not ideal, but it makes things a little more usable :)

vectorstorm commented 5 years ago

@jackhumbert That works as expected; the Planck EZ still freezes up if you press a key while USB is suspended, but it then restarts itself after a second or two, complete with startup audio and animating LEDs.

I've flashed back to my custom version of the firmware, which works around the issue by commenting out that call to osalThreadSleepMilliseconds() which I mentioned above. I expect that it breaks resume from keyboard, but as I don't ever put my computers to sleep (I fully power them off), that hasn't been a deal-breaker for me.

vectorstorm commented 5 years ago

Minor addition: if I comment out the setting and clearing of the USB_CNTR_RESUME flag on the lines above and below the sleep, and leave the sleep enabled, then that also completely avoids the freeze. So.. I'm now assuming that the sleep is just allowing another thread (or the hardware itself?) to pick up on that flag and try to do something that's freezing up the board; the sleep statement itself isn't actually the culprit.

Ciantic commented 5 years ago

This happens to me with Ergodox Ez too, I just upgraded my QMK Firmware. When I resume the computer from sleep (by pressing a key in keyboard), keyboard won't respond when coming to Windows and I have to unplug and plug it back again.

So the steps are:

Put computer to sleep
Press any key to wake up
Computer wakes up
All keys stop working
Unplug and plug back again and it works.

What is even worse is that I have another (unrelated) bug that requires me to flash each time I unplug and plug the keyboard. So I have to flash each time the computer comes from sleep :(

Ciantic commented 5 years ago

Apparently NO_USB_STARTUP_CHECK = yes makes it work, but it has a side effect: I can't wake my computer from sleep with keyboard anymore. I suspect I have to live with this.

It worked before without adding that flag, and it could wake up computer normally.

vectorstorm commented 5 years ago

@Ciantic It worked for you before? Do you happen to know what approximately what revision of the code you were running before? Might help me track down the issue

Ciantic commented 5 years ago

@vectorstorm I can't know the exact version, because I had deleted it already. It was either way probably ~6 months old branch or year old branch. Very old.

Ciantic commented 5 years ago

Should we close this? In favor of #5585 to not pollute the issues.

vectorstorm commented 5 years ago

See my comment above. My only issue with merging these two issues is that the workaround posted over there doesn’t work for me here, which makes me suspect they’re likely different issues, since they’re apparently freezing in different bits of code.

Ciantic commented 5 years ago

I'm pretty sure others who report in that issue has not tried that timeout workaround either, they just post that it won't work with them based on the parent post.

vectorstorm commented 5 years ago

For what it's worth, I've now tested right back to commit fa4052c26e33b42b65521fd5ba9f94a3a778b275, where the Planck EZ was first added to QMK, and I'm seeing the same freeze using the default keymap from that time. That commit was from April 8th of this year, and I can't go back any earlier, as it's the only QMK-enabled keyboard I have access to.

If it really is a bug that was added to QMK at some point (rather than, say, a toolchain change), then it must have happened before April 8th.

Ciantic commented 5 years ago

It's earlier. I made changes to my layout last time beginning of year, and before that 2018 july. Either of those I updated the branch.

stale[bot] commented 5 years ago

This issue has been automatically marked as resolved because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged properly or other activity occurs.

noroadsleft commented 5 years ago

:point_up: Teething issues with some new tooling we're trying to integrate.

drashna commented 4 years ago

@vectorstorm we have a potential fix for this issue. If you update your repo and reflash the board, does that fix the issue.

purxiz commented 4 years ago

@drashna I just downloaded the latest repo 2 days ago, and am still having the exact same issue described by the other people in this thread, i.e. board wakes up computer then freezes. I have to unplug/replug for it to work again.

drashna commented 4 years ago

Just curious, could you compile with CONSOLE_ENABLE=no and see if that helps?

Ciantic commented 4 years ago

If it helps, I tried this week ago with the latest Docker repo, didn't work for me either with Ergodox Ez.

Ledarium commented 4 years ago

Can confirm that with my handwired blackpill_f411 board. I compiled with CONSOLE_ENABLE=no and still having the issue. I am on latest qmk commit as for now.

callim105 commented 4 years ago

I also just stumbled upon this same issue, fwiw.

su8044 commented 4 years ago

Hey guys, this is what else i have found out so far:

on a split f401 board, i tried the idea of adding a 5000 ms wait mentioned https://github.com/qmk/qmk_firmware/issues/5585#issuecomment-491428389 but it didn't help
on a jj50 'NO_SUSPEND_POWER_DOWN = yes' does nothing to fix the sleep issue. it did introduce new issues to the rgb so i didn't test further
the most interesting: on the split, the first time i put the pc to sleep the rgb freezes as usual. if i wake by the power button then put the pc back to sleep the rgb turns off as expected. the keyboard freeze by touching any key still persists

tzarc commented 4 years ago

Can you try with #10088 applied?

su8044 commented 4 years ago

Update with #10088 applied:

NO MORE FREEZES AFTER WAKE! WOOHOOO

However, looks like the oleds won't turn off for the timeout of 60secs and the RGB and OLED animations stay frozen during sleep. Also, the keyboard looks frozen until i press 5 keys for it to respond during sleep, then it responds and the OLED info starts working correctly again. this behavior is only during sleep, once the pc wakes the keyboard is working as usual. not sure if i should post this info in #10088 too?

Edit: Before applying 10088 the rgb used to freeze too, but not the oleds, they would turn off after the timeout

XBowsTech commented 3 years ago

I also had this problem. Has a solution been found to this problem?

In addition, I found another solution, increasing the capacitance of the circuit board, can avoid this problem.

XBowsTech commented 3 years ago

For example, I used the is31fl3731 chip. I replaced the capacitor next to the chip with a 47uF capacitor, and the problem was solved. Although we did not understand the truth of the matter, this method did solve the problem.

spidey3 commented 3 years ago

Please try with #11450 applied...

drashna commented 1 month ago

There have been a number of significant fixes and improvements to the USB stack since this. I suspect that this is no longer occurring.

qmk / qmk_firmware