Open chipweinberger opened 2 years ago
From @Dazza0 from previous thread.
@chipweinberger Seems like a double HCD Port event. The hub only goes into the HUB_DRIVER_STATE_ROOT_RECOVERY state when the root port has an error (e.g., DISCONNECTION, ERROR, OVERCURRENT).
However, while hub driver is recovering the root port, there's another HCD_PORTEVENT... occurring, thus leading to the switch case calling the abort: Could you log what port_event is happening when the abort() is called?
Note: this abort happens when I unplug the USB device.
Do you know if this is happening every time you unplug the USB device? Usually, when you first unblock the USB device, a HCD_PORT_EVENT_DISCONNECTION event occurs which puts Hub driver into recovery. However, once disconnected, there shouldn't be any further HCD Port events.
So maybe there's an edge case missing where an HCD_PORT_EVENT_ERROR occurs first, followed rapidly by a HCD_PORT_EVENT_DISCONNECTION (it could happen the disconnection happens slowly).
@chipweinberger BTW, could you also open a separate Github issue to track this. I want to current issue to track the usb_transfer_t status feature mentioned above.
Answers:
is happening every time you unplug the USB device?
No. It's about 1 out of 20 unplugs (rough guess). Note: I sometimes plug and unplug very rapidly (trying to cause failures, for testing), but I can't remember if that is a prerequisite.
Could you log what port_event
Yes, I'll try this today. Is there a specific line of code to insert this log?
I finally repro'd it again!! It definitely takes awhile to repro (5+ minutes of trying). I was about to give up!
So the issue is: we are in HUB_DRIVER_STATE_ROOT_RECOVERY state, which is not handled by root_port_handle_events() switch statement, when I unplug the device.
First things first: I was only able to repro it with reinitAfterFailure=true
. Maybe with more trying I could repro it while it's false. I call this code whenever a USB transfer fails (for any reason). Pretty self explanatory.
// Not sure which one is more reliable...
// https://github.com/espressif/esp-idf/issues/8748
bool reinitAfterFailure = true;
if (reinitAfterFailure) {
// uninitializes everything
PDUsbHostService::uninitialize(error);
if (is_fail(*error)){
pd_raise_uie_e3("usbUninitialize", error);
}
} else {
// This will cause the midi-IN task to stop as well
PDUsbHostService::cancelAllTransfers();
// This will cause the usbh device to re-open
PDUsbHostService::closeAllDevices(error);
if (is_fail(*error)){
pd_raise_uie_e3("closeAllUsbDevices", error);
}
}
I added logs to port_event
& hal_port_evt
with these enum values, to be sure:
typedef enum {
USBH_HAL_PORT_EVENT_NONE = 0, /**< No event occurred, or could not decode interrupt */
USBH_HAL_PORT_EVENT_CHAN = 1, /**< A channel event has occurred. Call the the channel event handler instead */
USBH_HAL_PORT_EVENT_CONN = 2, /**< The host port has detected a connection */
USBH_HAL_PORT_EVENT_DISCONN = 3, /**< The host port has been disconnected */
USBH_HAL_PORT_EVENT_ENABLED = 4, /**< The host port has been enabled (i.e., connected to a device that has been reset. Started sending SOFs) */
USBH_HAL_PORT_EVENT_DISABLED = 5, /**< The host port has been disabled (no more SOFs). Could be due to disable/reset request, or a port error (e.g. port babble condition. See 11.8.1 of USB2.0 spec) */
USBH_HAL_PORT_EVENT_OVRCUR = 6, /**< The host port has encountered an overcurrent condition */
USBH_HAL_PORT_EVENT_OVRCUR_CLR = 7, /**< The host port has been cleared of the overcurrent condition */
} usbh_hal_port_event_t;
typedef enum {
HCD_PORT_EVENT_NONE = 0, /**< No event has occurred. Or the previous event is no longer valid */
HCD_PORT_EVENT_CONNECTION = 1, /**< A device has been connected to the port */
HCD_PORT_EVENT_DISCONNECTION = 2, /**< A device disconnection has been detected */
HCD_PORT_EVENT_ERROR = 3, /**< A port error has been detected. Port is now HCD_PORT_STATE_RECOVERY */
HCD_PORT_EVENT_OVERCURRENT = 4, /**< Overcurrent detected on the port. Port is now HCD_PORT_STATE_RECOVERY */
} hcd_port_event_t;
And this logging code to hcd.c
static void intr_hdlr_main(void *arg)
{
port_t *port = (port_t *) arg;
bool yield = false;
HCD_ENTER_CRITICAL_ISR();
usbh_hal_port_event_t hal_port_evt = usbh_hal_decode_intr(port->hal);
// log here
esp_rom_printf(DRAM_STR("hal:%u\n"), hal_port_evt);
if (hal_port_evt == USBH_HAL_PORT_EVENT_CHAN) {
//Channel event. Cycle through each pending channel
usbh_hal_chan_t *chan_obj = usbh_hal_get_chan_pending_intr(port->hal);
while (chan_obj != NULL) {
pipe_t *pipe = (pipe_t *)usbh_hal_chan_get_context(chan_obj);
hcd_pipe_event_t event = _intr_hdlr_chan(pipe, chan_obj, &yield);
//Run callback if a pipe event has occurred and the pipe also has a callback
if (event != HCD_PIPE_EVENT_NONE && pipe->callback != NULL) {
HCD_EXIT_CRITICAL_ISR();
yield |= pipe->callback((hcd_pipe_handle_t)pipe, event, pipe->callback_arg, true);
HCD_ENTER_CRITICAL_ISR();
}
//Check for more channels with pending interrupts. Returns NULL if there are no more
chan_obj = usbh_hal_get_chan_pending_intr(port->hal);
}
} else if (hal_port_evt != USBH_HAL_PORT_EVENT_NONE) { //Port event
hcd_port_event_t port_event = _intr_hdlr_hprt(port, hal_port_evt, &yield);
// log here
esp_rom_printf(DRAM_STR("port:%u\n"), port_event);
if (port_event != HCD_PORT_EVENT_NONE) {
port->last_event = port_event;
port->flags.event_pending = 1;
if (port->callback != NULL) {
HCD_EXIT_CRITICAL_ISR();
yield |= port->callback((hcd_port_handle_t)port, port_event, port->callback_arg, true);
HCD_ENTER_CRITICAL_ISR();
}
}
}
HCD_EXIT_CRITICAL_ISR();
if (yield) {
portYIELD_FROM_ISR();
}
}
Logs:
I (10834) pd usbh: usb_host_interface_claim(0x3deb84a0, deviceHandle:0x3deb85b0 bInterfaceNumber: 1 bAlternateSetting: 0)
I (10834) pd usbh: [host task] usb_host_lib_handle_events()
I (10854) pd midi usb: epAddrIn: 129
I (10864) pd midi usb IN: xTaskCreate xferInTask_loop
I (10864) pd midi usb: epAddrOut: 1
I (10864) pd log: allocated TLS for thread: xferInTask_loop. 0x3fcd8dcc.
I (10864) pd midi usb OUT: xTaskCreate midi_usb_tx_task
I (10864) pd midi usb IN: xferInTask_loop started
hal:1
I (10884) pd midi usb OUT: allocate msg queue: 44000 bytes
I (10894) pd midi usb IN: first Midi In. length: 64, 0x4a
W (10904) pd issues: RAISE - [Atypical] | usbMidiJunk
I (10904) pd_buzz: USB Device Inserted Sound
W (10914) pd midi usb IN: ignore junk. superhuman? no (15ms) junk? yesbuzz C 4 523Hz 100ms
W (10924) pd midi usb IN: 10.887 (usb open time)
junk: 10.902 | Cb:10 Ch:4 | Note Off G#14 vel:74
junk: 10.902 | Cb:1 Misc | M? 0x61 0xbd
junk: 10.902 | Cb:7 Misc | M? 0x50 0x15
junk: 10.902 | Cb:11 Misc | M? 0x69 0xae
junk: 10.902 | Cb:8 Sysex Start [3B] | 0x84 0xd4 0x54
junk: 10.902 | Cb:3 Ch:8 PolyKey Press | After Touch Poly E19 pressure:191
junk: 10.902 | Cb:2 Ch:2 PolyKey Press | After Touch Poly F#14 pressure:250
junk: 10.902 | Cb:7 Ch:13 Channel Pressure | After Touch Channel 212
junk: 10.902 | Cb:10 Ch:9 PolyKey Press | After Touch Poly F#15 pressure:254
junk: 10.902 | Cb:13 Sysex Start [3B] | 0x54 0x18 0x97
junk: 10.902 | Cb:3 Sysex End [1B] | Sysex Cin Packet 0xcd
junk: 10.902 | Cb:11 Ch:11 | Note Off G#4 vel:182
junk: 10.902 | Cb:15 Ch:4 Channel Pressure | After Touch Channel 241
junk: 10.902 | Cb:15 Ch:8 | Note On E4 vel:228
junk: 10.902 | Cb:2 Misc | M? 0x3a 0x2b
junk: 10.902 | Cb:5 Misc | M? 0x09 0x15
I (10904) pd log: allocated TLS for thread: midi_usb_tx. 0x3fcd8bb4.
I (10904) pd midi usb OUT: USB OUT task started
#
# I unplug the USB Device here
#
buzz F 4 698Hz 100ms
hal:3
port:2
I (11364) : e1 : [pd usbh] USB TRANSFER FAILED - no device
e2 : [pd usbh [device]] submitTransferAndWait() failed
e3 : [pd midi usb IN] xfer IN task: DEV GONE
I (11374) pd midi usb IN: xferInTask_loop ended. wait for usb replug.
I (11374) pd_buzz: USB Device Removed Sound
buzz F 4 698Hz 100ms
E (11364) USBH: Device 1 gone
W (11394) pd usbh: [client task] DEV_GONE dev_hdl: 0x3deb85b0
W (11404) pd usbh: [close device] closing, dev_hdl: 0x3deb85b0
I (11404) pd usbh: [close device] unblocking usb transfer 3
I (11414) pd usbh: usb_host_interface_release(0x3deb84a0, device: 0x3deb85b0, bInterfaceNumber: 1
I (11424) pd usbh: interface fully released
I (11424) pd usbh: [close device] usb_host_device_close(client:0x3deb84a0,dev:0x3deb85b0)
I (11444) pd usbh: [close device] 0 devices still open
I (11404) pd usbh: [host task] usb_host_lib_handle_events()
I (11444) pd usbh: [host task] event: USB_HOST_LIB_EVENT_FLAGS - ALL_FREE
I (11454) pd usbh: [host task] usb_host_lib_handle_events()
I (11454) pd usbh: [host task] usb_host_lib_handle_events()
buzz C 4 523Hz 100ms
I (11594) pd usbh: [uninitialize] called
I (11594) pd usbh: [uninitialize] got mutex
I (11594) pd usbh: cancelAllTransfers()
I (11594) pd usbh: [uninitialize] Stopping xUsbClientTask...
I (11604) pd usbh: [uninitialize] usb_host_client_unblock(client:0x3deb84a0)
I (11734) pd usbh: [client task] vTaskDelete()
I (12134) xesp usb parse: Free config pool
I (12134) pd usbh: [openSingleDevice] called
I (12134) pd usbh: [openSingleDevice] all devices closed. continuing...
I (12154) pd usbh: [openSingleDevice] initializing usbh
I (12154) pd usbh: [init] initialize
hal:2
port:1
I (12614) pd usbh: [uninitialize] xUsbClientTask completed
I (12614) pd usbh: [uninitialize] 0 devices need closing
I (12614) pd usbh: closeAllDevices()
I (12624) pd usbh: closeAllDevices() complete
I (12624) pd usbh: [uninitialize] usb_host_client_deregister(client:0x3deb84a0)
I (12634) pd usbh: [uninitialize] Stopping xUsbHostTask...
I (12634) pd usbh: [uninitialize] usb_host_lib_unblock()
hal:4
port:0
hal:1
hal:1
hal:1
I (12774) pd usbh: [host task] event: USB_HOST_LIB_EVENT_FLAGS - NO_CLIENTS
I (12774) pd usbh: [host task] usb_host_device_free_all()
I (12784) pd usbh: [host task] usb_host_lib_handle_events()
hal:5
port:0
hal:4
port:0
hal:1
hal:1
I (12844) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12844) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12854) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12854) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
I (12874) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12874) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12884) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12884) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12894) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12904) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12914) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12924) pd usbh: [host task] usb_host_lib_handle_events()
hal:1
hal:1
hal:1
I (12934) pd usbh: [host task] usb_host_lib_handle_events()
I (12944) pd usbh: [host task] usb_host_lib_handle_events()
I (12944) pd usbh: [host task] usb_host_lib_handle_events()
I (13064) pd usbh: [host task] vTaskDelete()
I (13654) pd usbh: [uninitialize] uxUsbHostTask completed
I (13654) pd usbh: [uninitialize] usb_host_uninstall()
process_pending_flags 0
lib_event_flags 0
flags.val 0
hal:3
port:2
ESP_ERROR_CHECK failed: esp_err_t 0x103Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC : 0x42087639 PS : 0x00060f33 A0 : 0x8208664d A1 : 0x3fca1d80
0x42087639: root_port_callback at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/usb/hub.c:703
A2 : 0x3fcd4bec A3 : 0x00000002 A4 : 0x00000000 A5 : 0x00000001
A6 : 0x00060023 A7 : 0x00060022 A8 : 0x3fcac374 A9 : 0x00000000
A10 : 0x00000001 A11 : 0xa23311f5 A12 : 0xa23311f5 A13 : 0x3fca79ac
A14 : 0x00060f23 A15 : 0x00060f22 SAR : 0x00000004 EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000000 LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000
Backtrace:0x42087636:0x3fca1d800x4208664a:0x3fca1db0 0x40378fc1:0x3fca1de0 0x420c50cc:0x3fcf5820 0x42006349:0x3fcf5850 0x403853eb:0x3fcf5880 0x40388252:0x3fcf58b0
0x42087636: root_port_callback at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/usb/hub.c:703
0x4208664a: intr_hdlr_main at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/usb/hcd.c:956
0x40378fc1: _xt_medint2 at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1200
0x420c50cc: cpu_ll_waiti at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/hal/esp32s3/include/hal/cpu_ll.h:182
(inlined by) esp_pm_impl_waiti at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esp_pm/pm_impl.c:837
0x42006349: esp_vApplicationIdleHook at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esp_system/freertos_hooks.c:63
0x403853eb: prvIdleTask at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/freertos/tasks.c:3973 (discriminator 1)
0x40388252: vPortTaskWrapper at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/freertos/port/xtensa/port.c:131
ELF file SHA256: 6ba9c090eb1d360d
I (10963) esp_core_dump_flash: Save core dump to flash...
I (10969) esp_core_dump_flash: Erase flash 53248 bytes @ 0x10000
I (11705) esp_core_dump_flash: Write end offset 0xc4a4, check sum length 4
I (11705) esp_core_dump_flash: Core dump has been saved to flash.
CPU halted.
bump =)
@Dazza0 , did you ever get a chance to reason about this?
Does root_port_handle_events()
need to handle HUB_DRIVER_STATE_ROOT_RECOVERY
?
Another bump!
bump! Have there been any changes around this?
bump again
did we ever get to the bottom of this?
I'm still hitting this on v5.1.2!
Crash when unplugging usb device
(10768) pd midi usb IN: 10.326 (usb open time)
junk: 10.328 | Cb:0 Ch:0 | Note Off A#4 vel:127
jmx-console> I (11568) pd usbh: wait for all devices to close...
E (11968) USBH: Device 1 gone
semaphore: 14
I (11968) : e1 : [pd usbh] USB TRANSFER FAILED - no device
e2 : [pd usbh [device]] pd_usbh_service_submitTransferAndWait() failed.
e3 : [pd midi usb IN] xfer IN task: DEV GONE
W (11968) pd usbh: [client task] DEV_GONE dev_hdl: 0x3c3b7100
I (11968) pd midi usb IN: xferInTask_loop ended. wait for usb replug.
I (11978) pd alerts: notify usb
I (11978) pd buzz: USB Device Removed Sound
W (11978) pd usbh: [close device] closing, dev_hdl: 0x3c3b7100
buzz F 4 698Hz 100ms
I (11978) pd usbh: [close device] unblocking usb transfer 3
I (11978) pd usbh: usb_host_interface_release(0x3c3b609c, device: 0x3c3b7100, bInterfaceNumber: 1
I (11988) pd usbh: interface fully released
I (11988) pd usbh: [close device] usb_host_device_close(client:0x3c3b609c,dev:0x3c3b7100)
I (11988) pd usbh: [close device] 0 devices still open
I (11968) pd usbh: [host task] usb_host_lib_handle_events()
abort() was called at PC 0x42090a07 on core 0
0x42090a07: root_port_handle_events at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/usb/hub.c:838
(inlined by) hub_process at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/usb/hub.c:1043
Backtrace: 0x40376037:0x3c3b5c00 0x40384b3d:0x3c3b5c30 0x4038d837:0x3c3b5c60 0x42090a07:0x3c3b5ce0 0x4208b49b:0x3c3b5d30 0x4204777f:0x3c3b5d70 0x40385aae:0x3c3b5da0
0x40376037: panic_abort at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esp_system/panic.c:452
0x40384b3d: esp_system_abort at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/esp_system/port/esp_system_chip.c:84
0x4038d837: abort at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/newlib/abort.c:38
0x42090a07: root_port_handle_events at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/usb/hub.c:838
(inlined by) hub_process at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/usb/hub.c:1043
0x4208b49b: usb_host_lib_handle_events at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/usb/usb_host.c:515
0x4204777f: usbHostTaskLoop(void*) at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/jamcorder-firmware/jcommon/usb/pd_usbh_service.cpp:70
0x40385aae: vPortTaskWrapper at /Volumes/User/MBP-Google-Drive/jamcorder/firmware/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:162
General issue report
ESP-IDF v4.4.2 ESP32-S3
See original issue: https://github.com/espressif/esp-idf/issues/8748 for more details.
Esp USB stack sometimes crashes (~5% chance) when unplugging a USB device.
The abort happens because we are in HUB_DRIVER_STATE_ROOT_RECOVERY state, which is not handled by
root_port_handle_events()
switch statement.Logs: