Open xnlcasad opened 1 year ago
Thanks for the report.
This is indeed different from earlier BSOD, which happened in VBoxUSB.sys. This one is in VBoxUSBMon.sys instead.
Normally, VBoxUSBMon is dormant until either a device is plugged/unplugged or attached/detached. When you say "at random times", is there any relation with USB devices changing state? Are there any USB filter drivers installed (usbipd-win list
would report those)?
In any case, it is an issue with the VBoxUSBMon driver, so you may want to report this to VirtualBox as well...
Report raised on their bugtracker I'll find the list of filter drivers as soon as I get access to the PCs that are showing this behaviour.
Just experienced a BSOD today on my hypervisor which I recently configured to run usbipd to share a Google Coral stick to a VM.
I did notice the application using Coral was regularly restarting due to losing the USB device.
The error from the below issue was also seen in my logs around the time Frigate crashed https://github.com/dorssel/usbipd-win/issues/423
Perhaps the constant resetting of the USB device resulted in the BSOD
@mateuszdrab Did you see which module / driver caused the BSOD? And what was the STOP code?
@dorssel sure, here's the entire windbg analyze output.
Microsoft (R) Windows Debugger Version 10.0.22621.755 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\temp\050924-46625-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available
Symbol search path is: srv*
Executable search path is:
Windows 10 Kernel Version 20348 MP (88 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Machine Name:
Kernel base = 0xfffff803`80200000 PsLoadedModuleList = 0xfffff803`80e33de0
Debug session time: Thu May 9 10:12:45.731 2024 (UTC + 1:00)
System Uptime: 10 days 18:45:29.032
Loading Kernel Symbols
...............................................................
................................................................
................................................................
........
Loading User Symbols
Loading unloaded module list
..................................................
For analysis of this file, run !analyze -v
0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
DPC_WATCHDOG_VIOLATION (133)
The DPC watchdog detected a prolonged run time at an IRQL of DISPATCH_LEVEL
or above.
Arguments:
Arg1: 0000000000000001, The system cumulatively spent an extended period of time at
DISPATCH_LEVEL or above. The offending component can usually be
identified with a stack trace.
Arg2: 0000000000001e00, The watchdog period.
Arg3: fffff80380f0f328, cast to nt!DPC_WATCHDOG_GLOBAL_TRIAGE_BLOCK, which contains
additional information regarding the cumulative timeout
Arg4: 0000000000000000
Debugging Details:
------------------
*** WARNING: Unable to verify timestamp for VBoxUSB.sys
*************************************************************************
*** ***
*** ***
*** Either you specified an unqualified symbol, or your debugger ***
*** doesn't have full symbol information. Unqualified symbol ***
*** resolution is turned off by default. Please either specify a ***
*** fully qualified symbol module!symbolname, or enable resolution ***
*** of unqualified symbols by typing ".symopt- 100". Note that ***
*** enabling unqualified symbol resolution with network symbol ***
*** server shares in the symbol path may cause the debugger to ***
*** appear to hang for long periods of time when an incorrect ***
*** symbol name is typed or the network symbol server is down. ***
*** ***
*** For some commands to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: TickPeriods ***
*** ***
*************************************************************************
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 3999
Key : Analysis.DebugAnalysisManager
Value: Create
Key : Analysis.Elapsed.mSec
Value: 4581
Key : Analysis.Init.CPU.mSec
Value: 1609
Key : Analysis.Init.Elapsed.mSec
Value: 19297
Key : Analysis.Memory.CommitPeak.Mb
Value: 122
FILE_IN_CAB: 050924-46625-01.dmp
DUMP_FILE_ATTRIBUTES: 0x808
Kernel Generated Triage Dump
BUGCHECK_CODE: 133
BUGCHECK_P1: 1
BUGCHECK_P2: 1e00
BUGCHECK_P3: fffff80380f0f328
BUGCHECK_P4: 0
DPC_TIMEOUT_TYPE: DPC_QUEUE_EXECUTION_TIMEOUT_EXCEEDED
BLACKBOXBSD: 1 (!blackboxbsd)
BLACKBOXNTFS: 1 (!blackboxntfs)
BLACKBOXPNP: 1 (!blackboxpnp)
BLACKBOXWINLOGON: 1
CUSTOMER_CRASH_COUNT: 1
PROCESS_NAME: usbipd.exe
STACK_TEXT:
fffff803`7ba8eda8 fffff803`804dd261 : 00000000`00000133 00000000`00000001 00000000`00001e00 fffff803`80f0f328 : nt!KeBugCheckEx
fffff803`7ba8edb0 fffff803`804db104 : 0007498c`d4a64f44 00006cfb`9de55318 ffff9304`621aad80 ffff9305`4c851502 : nt!KeAccumulateTicks+0x541
fffff803`7ba8ee20 fffff803`804dad6a : 00000000`29f59722 fffff803`7b7eb2e0 00000000`00000000 fffff803`804207bf : nt!KiUpdateRunTime+0x64
fffff803`7ba8eeb0 fffff803`804dabf4 : fffff803`80f01230 fffff803`80579efc fffff803`7b7e2180 00000000`00000000 : nt!KeClockInterruptNotify+0x10a
fffff803`7ba8ef40 fffff803`80579df0 : fffff803`81e4ef30 ffffa788`150b6000 ffff9305`41d295a8 fffff803`8062378f : nt!HalpTimerClockIpiRoutine+0x14
fffff803`7ba8ef70 fffff803`80623a0a : ffffa788`150b5e50 fffff803`80f01230 ffffa788`150b6000 ffff9304`62974da0 : nt!KiCallInterruptServiceRoutine+0xa0
fffff803`7ba8efb0 fffff803`806242d7 : 00000000`00000002 fffff803`80409b70 fffff803`81e4eb50 00000000`00000001 : nt!KiInterruptSubDispatchNoLockNoEtw+0xfa
ffffa788`150b5dd0 fffff803`81db3da5 : ffffffff`ffffffd2 fffff803`81e18f94 00000000`00000000 ffffa788`150b6020 : nt!KiInterruptDispatchNoLockNoEtw+0x37
ffffa788`150b5f60 fffff803`81df5417 : 00000000`00000000 00000000`00000001 00000000`00000000 ffffa788`150b6020 : Wdf01000!FxIrpQueue::PeekRequest+0x3d [minkernel\wdf\framework\shared\core\fxirpqueue.cpp @ 420]
ffffa788`150b5f90 fffff803`81dc1080 : ffff9304`68dd0960 ffff9304`68dd0960 00000000`025c0288 fffff803`81e4ef30 : Wdf01000!FxRequest::PeekRequest+0x27 [minkernel\wdf\framework\shared\core\fxrequest.cpp @ 2140]
ffffa788`150b5fc0 fffff803`81df4f7f : 00006cfa`da368f00 fffff803`9edc33b0 00000000`00000030 00006cfa`da368fd8 : Wdf01000!FxIoQueue::QueueIdle+0x170 [minkernel\wdf\framework\shared\irphandlers\io\fxioqueue.cpp @ 3713]
ffffa788`150b6050 fffff803`9ed149c2 : ffff9304`68dd0960 ffff9305`2a9859b0 00000000`ebe30031 00000000`00000000 : Wdf01000!imp_WdfIoQueueStopAndPurge+0x3f [minkernel\wdf\framework\shared\irphandlers\io\fxioqueueapi.cpp @ 541]
ffffa788`150b6080 fffff803`9edb03cd : 00000000`00000000 ffff9305`25c971f0 00000000`ebe30031 ffffa788`150b6740 : USBXHCI!Endpoint_UcxEvtEndpointAbort+0xc2
ffffa788`150b60e0 fffff803`9eda7560 : 00000000`ebe30031 ffff9305`4f16daf0 ffff9304`60d94720 fffff803`8042b35c : ucx01000!UcxEndpointStateEntryFunc_Aborting1+0x1d
ffffa788`150b6110 fffff803`9eda5dfe : ffff9305`4f16daf0 ffff9305`4f16daf0 ffff9305`25c971f0 ffff9304`60d94720 : ucx01000!StateMachineEngine_EventAdd+0x260
ffffa788`150b6160 fffff803`9eda32bb : ffff9304`60d94720 fffff803`9eda0000 00006cfb`9722f698 00006cfb`9b1f9698 : ucx01000!UrbHandler_AbortPipe+0x6e
ffffa788`150b6190 fffff803`9eda22a5 : fffff803`82d849b0 00000000`150b6740 fffff803`82d84901 ffff9304`60d94720 : ucx01000!Urb_USBPORTStyle_ProcessURB+0x7bb
ffffa788`150b6250 fffff803`81db5c60 : ffff9304`64e06960 00000000`00000600 00000000`00000000 00000000`00000001 : ucx01000!RootHub_Pdo_EvtInternalDeviceControlIrpPreprocessCallback+0xb5
ffffa788`150b62e0 fffff803`8055c2b5 : 00000000`00000004 ffff9304`650c7e10 00000000`00000000 ffff9305`4f16daf0 : Wdf01000!FxDevice::DispatchWithLock+0xd0 [minkernel\wdf\framework\shared\core\fxdevice.cpp @ 1447]
ffffa788`150b6340 fffff803`81e9af9d : ffff9304`650c7e10 ffff9305`4f16daf0 00000000`00000000 fffff803`80524fa9 : nt!IofCallDriver+0x55
ffffa788`150b6380 fffff803`81e910d6 : ffff9304`62180be0 00000000`00000007 ffff9305`4f16ded8 ffff9305`4c7f68a0 : ACPI!ACPIIrpDispatchDeviceControl+0xad
ffffa788`150b63c0 fffff803`8055c2b5 : 00000000`00000007 00000000`00000000 ffffa788`150b64e9 fffff803`9e527310 : ACPI!ACPIDispatchIrp+0xc6
ffffa788`150b6440 fffff803`9e4d75eb : ffff9305`4f16daf0 00000000`00000000 00000000`00000001 fffff803`80c494fe : nt!IofCallDriver+0x55
ffffa788`150b6480 fffff803`81db5c60 : ffff9305`1b62fdd0 ffff9305`4f16daf0 ffff9304`fd5388b0 ffff9305`4f16daf0 : UsbHub3!HUBPDO_EvtDeviceWdmIrpPreprocess+0x2bb
ffffa788`150b6550 fffff803`8055c2b5 : 00000000`00000000 ffff9305`050dfe10 00000000`00000000 ffff9305`4f16daf0 : Wdf01000!FxDevice::DispatchWithLock+0xd0 [minkernel\wdf\framework\shared\core\fxdevice.cpp @ 1447]
ffffa788`150b65b0 fffff803`81e9af9d : ffff9305`050dfe10 ffff9305`4f16daf0 00000000`0000000d 00000000`00000000 : nt!IofCallDriver+0x55
ffffa788`150b65f0 fffff803`81e910d6 : ffff9304`68d63bf0 00000000`00000007 ffff9305`4f16df20 ffff9304`5da02000 : ACPI!ACPIIrpDispatchDeviceControl+0xad
ffffa788`150b6630 fffff803`8055c2b5 : 00000000`00000007 ffff9305`050dfe10 00000000`ffffffff 00000000`00000000 : ACPI!ACPIDispatchIrp+0xc6
ffffa788`150b66b0 fffff803`a6f84355 : 00000000`00000002 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IofCallDriver+0x55
ffffa788`150b66f0 00000000`00000002 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : VBoxUSB+0x4355
ffffa788`150b66f8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x2
SYMBOL_NAME: USBXHCI!Endpoint_UcxEvtEndpointAbort+c2
MODULE_NAME: USBXHCI
IMAGE_NAME: USBXHCI.SYS
IMAGE_VERSION: 10.0.20348.1450
STACK_COMMAND: .cxr; .ecxr ; kb
BUCKET_ID_FUNC_OFFSET: c2
FAILURE_BUCKET_ID: 0x133_ISR_USBXHCI!Endpoint_UcxEvtEndpointAbort
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {0c82b582-a7fa-76d8-0eac-8fe9bb0424be}
Followup: MachineOwner
---------
That's the driver of the hub the device is connected to (USBXHCI.SYS
). Can you find that hub in the Windows Device Manager. Is it the stock Microsoft driver, or an OEM driver?
It looks like it is the stock driver.
I've now reshared the Coral Accelerator from a Pi 5 to a VM and it's been running much more stable.
There are no USB transfer error 4
errors logged by Frigate and it has not restarted for 20 hours.
When I was sharing from the Windows host, Frigate kept restarting every ~15 minutes due to that error.
@mateuszdrab Well, that's a bug in a Microsoft driver then. Better report it. Nothing I can do about that.
I will try to reproduce it on another system I don't mind BSODing, using both stock and Intel drivers and see if it makes any difference.
I've been following the issues raised before about Windows crashes with usbipd. See issues https://github.com/dorssel/usbipd-win/issues/461 https://github.com/dorssel/usbipd-win/issues/410 https://github.com/dorssel/usbipd-win/issues/248
It appears that https://github.com/dorssel/usbipd-win/commit/60d205dc36ac204597d5db4ef13034b6bf4c04eb commit and the 3.1.0 release was supposed to fix them.
We've been experiencing a different crash from that one, and we've seen in several Windows PCs. We are using usbipd 3.1.0
The crash happens at random times, we don't really know if there is a specific scenario that triggers it.
@dorssel Any ideas on how to find the root cause here ?
Below is the crash analysis from WinDBG.