search

google-coral / libedgetpu

Source code for the userspace level runtime driver for Coral.ai devices.
Apache License 2.0
172 stars 57 forks source link

Unknown: USB transfer error 1 [LibUsbDataInCallback] if process is kept running after inference. #41

Open waynieack opened 1 year ago

waynieack commented 1 year ago

Description

I have a Ubuntu 20.04.5 LTS VM with a full USB 3.0 card passed through to it with nothing but the Coral USB plugged into the card. I am able to load a model and run as many inference's with as many different images as I want with no issue, but if I try to hold the process in order to keep the model loaded so I can run more inference's later without loading the model again, the process always dies with the same Unknown: USB transfer error 1 error at the same exact spot in the debug logs.

I don't think this has anything to do with the cable or heat as it happens in the same exact point in the debug logs and I can reproduce it in the with the same exact process every time.

I made a very simple way to reproduce this by adding a time.sleep(300) at the bottom of the examples/classify_image.py example script in the pycoral repo. See the full debug below:

Click to expand! ### Issue Type _No response_ ### Operating System _No response_ ### Coral Device _No response_ ### Other Devices _No response_ ### Programming Language _No response_ ### Relevant Log Output ```shell /coral/pycoral# sudo -u www-data python3 examples/classify_image.py --model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite --labels test_data/inat_bird_labels.txt --input test_data/parrot.jpg I tflite/edgetpu_manager_direct.cc:453] No matching device is already opened for shared ownership. I driver/driver_factory_default.cc:31] Failed to open /sys/class/apex: No such file or directory I driver/usb/local_usb_device.cc:944] EnumerateDevices: vendor:0x1a6e, product:0x89a I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[2] port[1] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[2] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[8] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[7] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[6] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[9] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[10] port[2] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[10] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[1] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[5] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[4] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[3] port[0] I driver/usb/local_usb_device.cc:944] EnumerateDevices: vendor:0x18d1, product:0x9302 I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[2] port[1] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[2] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[8] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[7] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[6] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[9] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[10] port[2] I driver/usb/local_usb_device.cc:998] EnumerateDevices: found [/sys/bus/usb/devices/10-2] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[10] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[1] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[5] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[4] port[0] I driver/usb/local_usb_device.cc:979] EnumerateDevices: checking bus[3] port[0] I driver/beagle/beagle_usb_driver_provider.cc:225] Enumerate: adding path [/sys/bus/usb/devices/10-2] I tflite/edgetpu_manager_direct.cc:471] No device of type Apex (PCIe) is available. I tflite/edgetpu_context_direct.cc:106] USB always DFU: False (default) I tflite/edgetpu_context_direct.cc:147] USB bulk-in queue capacity: 8 I tflite/edgetpu_context_direct.cc:67] Performance expectation: Max (default) I driver/usb/usb_driver.cc:1383] Open device and check if DFU is needed I driver/usb/local_usb_device.cc:1013] OpenDevice: [/sys/bus/usb/devices/10-2] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[2] port[1] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[2] port[0] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[8] port[0] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[7] port[0] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[6] port[0] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[9] port[0] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[10] port[2] I driver/usb/local_usb_device.cc:1081] OpenDevice: device opened 0x2d7a500 I driver/usb/local_usb_device.cc:184] LocalUsbDevice I driver/usb/usb_standard_commands.cc:36] UsbStandardCommands I driver/usb/usb_dfu_commands.cc:37] UsbDfuCommands I driver/usb/usb_standard_commands.cc:43] GetDeviceDescriptor I driver/usb/local_usb_device.cc:398] GetDescriptor I driver/usb/usb_standard_commands.cc:78] Vender ID: 0x18d1 I driver/usb/usb_standard_commands.cc:79] Product ID: 0x9302 I driver/usb/usb_driver.cc:1410] Device is already in application mode, skipping DFU I driver/usb/usb_driver.cc:1422] Resetting device I driver/usb/local_usb_device.cc:243] Close: closing device 0x2d7a500 I driver/usb/local_usb_device.cc:216] DoCancelAllTransfers: cancelling 0 async transfers I driver/usb/local_usb_device.cc:224] DoCancelAllTransfers: waiting for all async transfers to complete I driver/usb/local_usb_device.cc:234] DoCancelAllTransfers: all async transfers have completed I driver/usb/local_usb_device.cc:276] Close: releasing 0 transfer buffers I driver/usb/local_usb_device.cc:289] Close: performing graceful reset I driver/usb/local_usb_device.cc:322] Close: final clean up completed I driver/usb/usb_driver.cc:1364] Opening device expecting application mode I driver/usb/local_usb_device.cc:1013] OpenDevice: [/sys/bus/usb/devices/10-2] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[2] port[1] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[2] port[0] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[8] port[0] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[7] port[0] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[6] port[0] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[9] port[0] I driver/usb/local_usb_device.cc:1050] OpenDevice: checking bus[10] port[2] I driver/usb/local_usb_device.cc:1081] OpenDevice: device opened 0x2e0ebd0 I driver/usb/local_usb_device.cc:184] LocalUsbDevice I driver/usb/usb_standard_commands.cc:36] UsbStandardCommands I driver/usb/usb_ml_commands.cc:47] UsbMlCommands I driver/usb/usb_dfu_commands.cc:40] ~UsbDfuCommands I driver/usb/usb_standard_commands.cc:39] ~UsbStandardCommands I driver/usb/local_usb_device.cc:196] ~LocalUsbDevice I driver/usb/local_usb_device.cc:243] Close: closing device (nil) I driver/usb/local_usb_device.cc:352] ClaimInterface I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a30c I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A30C] == 0xF0059 I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A30C] := 0xF0059 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a314 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A314] == 0x110000 I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a318 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A318] == 0x850004 I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A318] := 0xC50004 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a318 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A318] == 0xC5025C I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1907C] := 0xF I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1907C] := 0x0 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a318 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A318] == 0xC5025C I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A318] := 0x85025C I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a318 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A318] == 0x850004 I driver/usb/usb_ml_commands.cc:117] ReadRegister64 offset 0x44018 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:147] ReadRegister64 [0x44018] == 0x0 I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x4A000] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x48788] := 0x7F I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:117] ReadRegister64 offset 0x48788 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:147] ReadRegister64 [0x48788] == 0x7F I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x40020] := 0x1E02 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a314 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A314] == 0x110000 I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A314] := 0x150000 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a000 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A000] == 0x219089A I driver/usb/usb_driver.cc:321] e-fuse programming revision: 2 I driver/usb/usb_driver.cc:328] InitializeChip Enabling only sc host interrupt descriptors I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x4C148] := 0xF0 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_driver.cc:341] InitializeChip Enabling single EP mode I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x4C160] := 0x0 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_driver.cc:369] InitializeChip Setting 1KB chunk for bulk-ins I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x4C058] := 0x80 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x44018] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x44158] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x44198] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x441D8] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x44218] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x48788] := 0x7F I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:117] ReadRegister64 offset 0x48788 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:147] ReadRegister64 [0x48788] == 0x7F I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x400C0] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x40150] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x40110] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x40250] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x40298] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x402E0] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x40328] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x40190] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x401D0] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x40210] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x4C060] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x4C070] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x4C080] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x4C090] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:178] WriteRegister64 [0x4C0A0] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a0d4 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A0D4] == 0x80000001 I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A0D4] := 0x80000001 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a704 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A704] == 0x7F I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A704] := 0x7F I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a33c I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A33C] == 0x3F I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A33C] := 0x3F I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A500] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A600] := 0x1 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A558] := 0x3 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A658] := 0x3 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I driver/usb/usb_ml_commands.cc:80] ReadRegister32 offset 0x1a0d8 I driver/usb/local_usb_device.cc:514] SendControlCommandWithDataIn I driver/usb/local_usb_device.cc:521] SYNC CTRL WITH DATA IN begin I driver/usb/local_usb_device.cc:538] SYNC CTRL WITH DATA IN end I driver/usb/usb_ml_commands.cc:110] ReadRegister32 [0x1A0D8] == 0x80000000 I driver/usb/usb_ml_commands.cc:153] WriteRegister32 [0x1A0D8] := 0x80000000 I driver/usb/local_usb_device.cc:475] SendControlCommandWithDataOut I driver/usb/local_usb_device.cc:482] SYNC CTRL WITH DATA OUT begin I driver/usb/local_usb_device.cc:499] SYNC CTRL WITH DATA OUT end I tflite/edgetpu_context_direct.cc:174] Opening device at /sys/bus/usb/devices/10-2 I driver/usb/usb_driver.cc:1155] WorkerThreadFunc starting worker thread I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 0 callback events in worker thread I driver/usb/usb_driver.cc:1210] WorkerThreadFunc Re-installing event reader I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 2 begin I driver/usb/usb_driver.cc:1232] WorkerThreadFunc Re-installing interrupt reader I driver/usb/local_usb_device.cc:785] AsyncInterruptInTransfer I driver/usb/local_usb_device.cc:798] ASYNC IN 3 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [0] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [1] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [2] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [3] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [4] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [5] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [6] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [7] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1317] WorkerThreadFunc waiting on state change I driver/usb/usb_driver.cc:91] Unlocks both mutex examples/classify_image.py:81: DeprecationWarning: ANTIALIAS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead. image = Image.open(args.input).convert('RGB').resize(size, Image.ANTIALIAS) ----INFERENCE TIME---- Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory. I driver/request.cc:47] Adding input "map/TensorArrayStack/TensorArrayGatherV3" with 150528 bytes. I driver/request.cc:58] Adding output "prediction" with 965 bytes. I driver/request.cc:167] Request prepared, total batch size: 1, total TPU requests required: 1. I driver/driver.cc:307] Request [0]: Submitting P0 request immediately. I driver/driver.cc:369] Request [0]: Need to map parameters. I driver/driver.cc:249] Mapped params : Buffer(ptr=0x7f4494133000) -> 0x00007f4494133000, 3947392 bytes. I driver/driver.cc:249] Mapped params : Buffer(ptr=(nil)) -> 0x0000000000000000, 0 bytes. I driver/driver.cc:383] Request [0]: Need to do parameter-caching. I driver/single_tpu_request.cc:80] [0] Request constructed. I driver/instruction_buffers.cc:46] InstructionBuffers created. I driver/package_registry.cc:647] Created new instruction buffers. I driver/device_buffer_mapper.cc:75] Mapped scratch : Buffer(ptr=(nil)) -> 0x0000000000000000, 0 bytes. I driver/single_tpu_request.cc:365] MapDataBuffers() done. I driver/executable_util.cc:187] Linking Parameter: 0x00007f4494133000 I driver/device_buffer_mapper.cc:222] Mapped "instructions" : Buffer(ptr=0x2e6f000) -> 0x0000000002e6f000, 10064 bytes. Direction=1 I driver/single_tpu_request.cc:381] MapInstructionBuffers() done. I driver/single_tpu_request.cc:478] [0] SetState old=0, new=1. I driver/single_tpu_request.cc:390] [0] NotifyRequestSubmitted() I driver/single_tpu_request.cc:478] [0] SetState old=1, new=2. I driver/single_queue_dma_scheduler.cc:82] Request[0]: Submitted I driver/dma_info_extractor.cc:166] DMA[0]: Instruction: device_address = 0x2e6f000, bytes = 10064, pending I driver/dma_info_extractor.cc:166] DMA[1]: Parameter: device_address = 0x7f4494133000, bytes = 3947392, pending I driver/dma_info_extractor.cc:166] DMA[2]: SC interrupt 0 I driver/single_tpu_request.cc:80] [1] Request constructed. I driver/single_tpu_request.cc:113] Adding input "map/TensorArrayStack/TensorArrayGatherV3" with 150528 bytes. I driver/single_tpu_request.cc:187] Adding output "prediction" with 965 bytes. I driver/usb/usb_driver.cc:86] lock (does nothing) I driver/usb/usb_driver.cc:1322] WorkerThreadFunc driver state change detected I driver/instruction_buffers.cc:46] InstructionBuffers created. I driver/package_registry.cc:647] Created new instruction buffers. I driver/device_buffer_mapper.cc:75] Mapped scratch : Buffer(ptr=(nil)) -> 0x0000000000000000, 0 bytes. I driver/device_buffer_mapper.cc:222] Mapped "map/TensorArrayStack/TensorArrayGatherV3" : Buffer(ptr=0x2e22d40) -> 0x0000000002e22d40, 150528 bytes. Direction=1 I driver/device_buffer_mapper.cc:222] Mapped "prediction" : Buffer(ptr=0x2e74000) -> 0x0000000002e74000, 968 bytes. Direction=2 I driver/single_tpu_request.cc:365] MapDataBuffers() done. I driver/executable_util.cc:93] Linking map/TensorArrayStack/TensorArrayGatherV3[0]: 0x0000000002e22d40 I driver/executable_util.cc:93] Linking prediction[0]: 0x0000000002e74000 I driver/device_buffer_mapper.cc:222] Mapped "instructions" : Buffer(ptr=0x2e76000) -> 0x0000000002e76000, 261920 bytes. Direction=1 I driver/device_buffer_mapper.cc:222] Mapped "instructions" : Buffer(ptr=0x2eb7000) -> 0x0000000002eb7000, 10224 bytes. Direction=1 I driver/single_tpu_request.cc:381] MapInstructionBuffers() done. I driver/single_tpu_request.cc:478] [1] SetState old=0, new=1. I driver/single_tpu_request.cc:390] [1] NotifyRequestSubmitted() I driver/single_tpu_request.cc:478] [1] SetState old=1, new=2. I driver/single_queue_dma_scheduler.cc:82] Request[1]: Submitted I driver/dma_info_extractor.cc:166] DMA[0]: Instruction: device_address = 0x2e76000, bytes = 261920, pending I driver/dma_info_extractor.cc:166] DMA[1]: Input activation: device_address = 0x2e22d40, bytes = 150528, pending I driver/dma_info_extractor.cc:166] DMA[2]: Instruction: device_address = 0x2eb7000, bytes = 10224, pending I driver/dma_info_extractor.cc:166] DMA[3]: Output activation: device_address = 0x2e74000, bytes = 968, pending I driver/dma_info_extractor.cc:166] DMA[4]: SC interrupt 0 I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 0 callback events in worker thread I driver/single_tpu_request.cc:398] [0] NotifyRequestActive() I driver/single_tpu_request.cc:478] [0] SetState old=2, new=3. I driver/single_queue_dma_scheduler.cc:132] Request[0]: Scheduling DMA[0] I driver/single_queue_dma_scheduler.cc:132] Request[0]: Scheduling DMA[1] I driver/single_queue_dma_scheduler.cc:132] Request[0]: Scheduling DMA[2] I driver/single_tpu_request.cc:398] [1] NotifyRequestActive() I driver/single_tpu_request.cc:478] [1] SetState old=2, new=3. I driver/single_queue_dma_scheduler.cc:132] Request[1]: Scheduling DMA[0] I driver/single_queue_dma_scheduler.cc:132] Request[1]: Scheduling DMA[1] I driver/single_queue_dma_scheduler.cc:132] Request[1]: Scheduling DMA[2] I driver/single_queue_dma_scheduler.cc:132] Request[1]: Scheduling DMA[3] I driver/single_queue_dma_scheduler.cc:132] Request[1]: Scheduling DMA[4] I driver/usb/usb_driver.cc:777] ProcessIo [0-0] bulk out header I driver/usb/usb_ml_commands.cc:215] PrepareHeader ep 1: header hex 50 27 0 0 - 0 0 0 0 I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/dma_chunker.cc:42] Completed 0 bytes; Outstanding 0 bytes; Processing next 10064 bytes I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/usb/usb_driver.cc:777] ProcessIo [1-2] bulk out header I driver/usb/usb_ml_commands.cc:215] PrepareHeader ep 1: header hex 80 3b 3c 0 - 2 0 0 0 I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/dma_chunker.cc:42] Completed 0 bytes; Outstanding 0 bytes; Processing next 1048576 bytes I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/usb_driver.cc:1310] WorkerThreadFunc re-evaluation is needed I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 3 callback events in worker thread I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:796] [0-0] bulk out for header done I driver/usb/usb_driver.cc:850] operator() [0-0] bulk out for 10064 bytes done I driver/usb/usb_driver.cc:796] [1-2] bulk out for header done I driver/single_queue_dma_scheduler.cc:154] Completing DMA[0] I driver/usb/usb_driver.cc:550] IO completed I driver/dma_chunker.cc:42] Completed 0 bytes; Outstanding 1048576 bytes; Processing next 1048576 bytes I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/usb/usb_driver.cc:1310] WorkerThreadFunc re-evaluation is needed I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 0 callback events in worker thread I driver/dma_chunker.cc:42] Completed 0 bytes; Outstanding 2097152 bytes; Processing next 1048576 bytes I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/usb/usb_driver.cc:1310] WorkerThreadFunc re-evaluation is needed I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 0 callback events in worker thread I driver/usb/usb_driver.cc:643] [1-2] number of concurrent transfers too high, wait (3 >= 3) I driver/usb/usb_driver.cc:1317] WorkerThreadFunc waiting on state change I driver/usb/usb_driver.cc:91] Unlocks both mutex I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:86] lock (does nothing) I driver/usb/usb_driver.cc:1322] WorkerThreadFunc driver state change detected I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 1 callback events in worker thread I driver/usb/usb_driver.cc:850] operator() [1-2] bulk out for 1048576 bytes done I driver/dma_chunker.cc:42] Completed 1048576 bytes; Outstanding 2097152 bytes; Processing next 801664 bytes I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/usb/usb_driver.cc:643] [0-0] number of concurrent transfers too high, wait (3 >= 3) I driver/usb/usb_driver.cc:1310] WorkerThreadFunc re-evaluation is needed I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 0 callback events in worker thread I driver/usb/usb_driver.cc:643] [1-2] number of concurrent transfers too high, wait (3 >= 3) I driver/usb/usb_driver.cc:1317] WorkerThreadFunc waiting on state change I driver/usb/usb_driver.cc:91] Unlocks both mutex I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:86] lock (does nothing) I driver/usb/usb_driver.cc:1322] WorkerThreadFunc driver state change detected I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 1 callback events in worker thread I driver/usb/usb_driver.cc:850] operator() [1-2] bulk out for 1048576 bytes done I driver/usb/usb_driver.cc:777] ProcessIo [0-0] bulk out header I driver/usb/usb_ml_commands.cc:215] PrepareHeader ep 1: header hex 20 ff 3 0 - 0 0 0 0 I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/dma_chunker.cc:42] Completed 0 bytes; Outstanding 0 bytes; Processing next 261920 bytes I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/usb/usb_driver.cc:643] [1-1] number of concurrent transfers too high, wait (4 >= 3) I driver/usb/usb_driver.cc:1310] WorkerThreadFunc re-evaluation is needed I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 0 callback events in worker thread I driver/usb/usb_driver.cc:643] [0-0] number of concurrent transfers too high, wait (3 >= 3) I driver/usb/usb_driver.cc:1317] WorkerThreadFunc waiting on state change I driver/usb/usb_driver.cc:91] Unlocks both mutex I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:86] lock (does nothing) I driver/usb/usb_driver.cc:1322] WorkerThreadFunc driver state change detected I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 1 callback events in worker thread I driver/usb/usb_driver.cc:850] operator() [1-2] bulk out for 1048576 bytes done I driver/usb/usb_driver.cc:777] ProcessIo [1-1] bulk out header I driver/usb/usb_ml_commands.cc:215] PrepareHeader ep 1: header hex 0 4c 2 0 - 1 0 0 0 I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/dma_chunker.cc:42] Completed 0 bytes; Outstanding 0 bytes; Processing next 150528 bytes I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/usb/usb_driver.cc:643] [2-0] number of concurrent transfers too high, wait (4 >= 3) I driver/usb/usb_driver.cc:1310] WorkerThreadFunc re-evaluation is needed I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 0 callback events in worker thread I driver/usb/usb_driver.cc:643] [1-1] number of concurrent transfers too high, wait (3 >= 3) I driver/usb/usb_driver.cc:1317] WorkerThreadFunc waiting on state change I driver/usb/usb_driver.cc:91] Unlocks both mutex I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:672] ASYNC IN 2 end I driver/usb/usb_ml_commands.cc:258] operator() tag:4, offset:0x0, length 0 I driver/usb/usb_ml_commands.cc:266] operator() callback done I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:86] lock (does nothing) I driver/usb/usb_driver.cc:1322] WorkerThreadFunc driver state change detected I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 3 callback events in worker thread I driver/usb/usb_driver.cc:850] operator() [1-2] bulk out for 801664 bytes done I driver/usb/usb_driver.cc:796] [0-0] bulk out for header done I driver/usb/usb_driver.cc:1050] Digesting descriptor from device tag[4], data[0x0], size[0] I driver/usb/usb_io_request.cc:94] DMA[2] hint matched with descriptor I driver/usb/usb_driver.cc:1210] WorkerThreadFunc Re-installing event reader I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 2 begin I driver/single_queue_dma_scheduler.cc:154] Completing DMA[1] I driver/usb/usb_driver.cc:550] IO completed I driver/single_queue_dma_scheduler.cc:154] Completing DMA[2] I driver/single_tpu_request.cc:410] [0] NotifyCompletion() I driver/package_registry.cc:658] Returned instruction buffers back to executable reference I driver/single_tpu_request.cc:478] [0] SetState old=3, new=4. I driver/single_queue_dma_scheduler.cc:234] Request[0]: Completed I driver/single_tpu_request.cc:96] [0] Request destroyed. I driver/usb/usb_driver.cc:550] IO completed I driver/usb/usb_driver.cc:777] ProcessIo [2-0] bulk out header I driver/usb/usb_ml_commands.cc:215] PrepareHeader ep 1: header hex f0 27 0 0 - 0 0 0 0 I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/dma_chunker.cc:42] Completed 0 bytes; Outstanding 0 bytes; Processing next 10224 bytes I driver/usb/local_usb_device.cc:708] AsyncBulkOutTransfer I driver/usb/local_usb_device.cc:722] ASYNC OUT 1 begin I driver/usb/usb_driver.cc:1310] WorkerThreadFunc re-evaluation is needed I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 0 callback events in worker thread I driver/usb/usb_driver.cc:643] [2-0] number of concurrent transfers too high, wait (3 >= 3) I driver/usb/usb_driver.cc:1317] WorkerThreadFunc waiting on state change I driver/usb/usb_driver.cc:91] Unlocks both mutex I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:86] lock (does nothing) I driver/usb/usb_driver.cc:1322] WorkerThreadFunc driver state change detected I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 2 callback events in worker thread I driver/usb/usb_driver.cc:850] operator() [0-0] bulk out for 261920 bytes done I driver/usb/usb_driver.cc:796] [1-1] bulk out for header done I driver/single_queue_dma_scheduler.cc:154] Completing DMA[0] I driver/usb/usb_driver.cc:550] IO completed I driver/usb/usb_driver.cc:1317] WorkerThreadFunc waiting on state change I driver/usb/usb_driver.cc:91] Unlocks both mutex I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:657] ASYNC OUT 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:86] lock (does nothing) I driver/usb/usb_driver.cc:1322] WorkerThreadFunc driver state change detected I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 3 callback events in worker thread I driver/usb/usb_driver.cc:850] operator() [1-1] bulk out for 150528 bytes done I driver/usb/usb_driver.cc:796] [2-0] bulk out for header done I driver/usb/usb_driver.cc:850] operator() [2-0] bulk out for 10224 bytes done I driver/single_queue_dma_scheduler.cc:154] Completing DMA[1] I driver/usb/usb_driver.cc:550] IO completed I driver/single_queue_dma_scheduler.cc:154] Completing DMA[2] I driver/usb/usb_driver.cc:550] IO completed I driver/usb/usb_driver.cc:1317] WorkerThreadFunc waiting on state change I driver/usb/usb_driver.cc:91] Unlocks both mutex I driver/usb/local_usb_device.cc:672] ASYNC IN 1 end I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:672] ASYNC IN 2 end I driver/usb/usb_ml_commands.cc:258] operator() tag:4, offset:0x0, length 0 I driver/usb/usb_ml_commands.cc:266] operator() callback done I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:86] lock (does nothing) I driver/usb/usb_driver.cc:1322] WorkerThreadFunc driver state change detected I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 2 callback events in worker thread I driver/usb/usb_driver.cc:1138] bulk in 968 bytes from buffer index [0] I driver/usb/usb_driver.cc:1050] Digesting descriptor from device tag[4], data[0x0], size[0] I driver/usb/usb_io_request.cc:94] DMA[4] hint matched with descriptor I driver/usb/usb_driver.cc:1210] WorkerThreadFunc Re-installing event reader I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 2 begin I driver/dma_chunker.cc:30] Completed 0 bytes; Outstanding 0 bytes; Processing next 968 bytes I driver/usb/usb_driver.cc:916] [3-3] bulk in for 968 bytes has yielded 968 bytes from index [0] I driver/usb/usb_driver.cc:1310] WorkerThreadFunc re-evaluation is needed I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 0 callback events in worker thread I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [0] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/single_queue_dma_scheduler.cc:154] Completing DMA[3] I driver/usb/usb_driver.cc:550] IO completed I driver/single_queue_dma_scheduler.cc:154] Completing DMA[4] I driver/single_tpu_request.cc:410] [1] NotifyCompletion() I driver/package_registry.cc:658] Returned instruction buffers back to executable reference I driver/single_tpu_request.cc:478] [1] SetState old=3, new=4. I driver/single_queue_dma_scheduler.cc:234] Request[1]: Completed I driver/single_tpu_request.cc:96] [1] Request destroyed. I driver/usb/usb_driver.cc:550] IO completed I driver/usb/usb_driver.cc:1317] WorkerThreadFunc waiting on state change I driver/usb/usb_driver.cc:91] Unlocks both mutex 16.7ms -------RESULTS-------- Ara macao (Scarlet Macaw): 0.75781 I driver/usb/local_usb_device.cc:672] ASYNC IN 3 end I driver/usb/local_usb_device.cc:97] ConvertLibUsbTransferStatus: USB transfer error 2 [LibUsbDataInCallback] I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:672] ASYNC IN 1 end I driver/usb/local_usb_device.cc:97] ConvertLibUsbTransferStatus: USB transfer error 2 [LibUsbDataInCallback] I driver/usb/usb_driver.cc:86] lock (does nothing) I driver/usb/usb_driver.cc:1322] WorkerThreadFunc driver state change detected I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 2 callback events in worker thread I driver/usb/usb_driver.cc:468] USB transfer error 2 [LibUsbDataInCallback] I driver/usb/usb_driver.cc:1232] WorkerThreadFunc Re-installing interrupt reader I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:785] AsyncInterruptInTransfer I driver/usb/local_usb_device.cc:798] ASYNC IN 3 begin I driver/usb/local_usb_device.cc:672] ASYNC IN 1 end I driver/usb/local_usb_device.cc:97] ConvertLibUsbTransferStatus: USB transfer error 2 [LibUsbDataInCallback] I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:672] ASYNC IN 1 end I driver/usb/local_usb_device.cc:97] ConvertLibUsbTransferStatus: USB transfer error 2 [LibUsbDataInCallback] I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [1] I driver/usb/local_usb_device.cc:672] ASYNC IN 1 end I driver/usb/local_usb_device.cc:97] ConvertLibUsbTransferStatus: USB transfer error 2 [LibUsbDataInCallback] I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:672] ASYNC IN 1 end I driver/usb/local_usb_device.cc:97] ConvertLibUsbTransferStatus: USB transfer error 2 [LibUsbDataInCallback] I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:672] ASYNC IN 1 end I driver/usb/local_usb_device.cc:97] ConvertLibUsbTransferStatus: USB transfer error 2 [LibUsbDataInCallback] I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1324] WorkerThreadFunc callback event available. skip waiting I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 5 callback events in worker thread I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [2] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:672] ASYNC IN 1 end I driver/usb/local_usb_device.cc:97] ConvertLibUsbTransferStatus: USB transfer error 2 [LibUsbDataInCallback] I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [3] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [4] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [5] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [6] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1324] WorkerThreadFunc callback event available. skip waiting I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 1 callback events in worker thread I driver/usb/usb_driver.cc:1260] WorkerThreadFunc Installing bulk-in reader. buffer index [7] I driver/usb/local_usb_device.cc:748] AsyncBulkInTransfer I driver/usb/local_usb_device.cc:761] ASYNC IN 1 begin I driver/usb/usb_driver.cc:1317] WorkerThreadFunc waiting on state change I driver/usb/usb_driver.cc:91] Unlocks both mutex I driver/usb/local_usb_device.cc:672] ASYNC IN 1 end I driver/usb/local_usb_device.cc:97] ConvertLibUsbTransferStatus: USB transfer error 1 [LibUsbDataInCallback] I driver/usb/local_usb_device.cc:642] UnregisterCompletedTransfer I driver/usb/usb_driver.cc:86] lock (does nothing) I driver/usb/usb_driver.cc:1322] WorkerThreadFunc driver state change detected I driver/usb/usb_driver.cc:1174] WorkerThreadFunc dispatching 1 callback events in worker thread F driver/usb/usb_driver.cc:1148] HandleQueuedBulkIn transfer in failed. Unknown: USB transfer error 1 [LibUsbDataInCallback] Aborted ```
waynieack commented 1 year ago

Kernel Info: 5.15.0-56-generic #62~20.04.1-Ubuntu SMP

CPU Info: /coral/pycoral# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 40 bits physical, 48 bits virtual CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 8 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz Stepping: 2 CPU MHz: 2493.996 BogoMIPS: 4987.99 L1d cache: 256 KiB L1i cache: 256 KiB L2 cache: 32 MiB L3 cache: 16 MiB NUMA node0 CPU(s): 0-7 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopolog y cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti fsgsba se tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat umip arch_capabilities

waynieack commented 1 year ago

Also as long as I don't run an inference, I can load the model and sleep the script for as long as I want without the error happening. As soon as a inference is run and the sleep starts, it will crash with the same error every time. If I run the inference in a loop, as long as they are running, it will not crash.

This is the script with the sleeps before and after the inference. https://gist.github.com/waynieack/b7002a89b49e72c3af0947e778509f0f

This is the logs from running the script above. https://gist.github.com/waynieack/1a4e74cfa4d9811ab8ab4bc639cc1f75

hjonnala commented 1 year ago

The issue is not happenning on my Linux machine. Are you running on a virtual machine? If so, please try the same thing on native OS.

:uname -a
Linux hemanth 5.19.11-1rodete1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.19.11-1rodete1 (2022-10-31) x86_64 GNU/Linux
waynieack commented 1 year ago

I don't have a physical machine with the same hardware as the host, in addition the kernel version is different. So even if it works on the physical machine, it doesn't tell me much because there's a bunch of other differences.

Regardless, I need it to work on the virtual machine not a physical machine. The issue doesn't seem to be related to throughput because its not even processing anything. I am passing through the full USB 3.0 PCI card per what I've read in the forums.

waynieack commented 1 year ago

I moved it to a port on an onboard controller and just passed the port through and I don't have the issue anymore. I tried the card with just port pass through instead of full PCI pass through and I have the same issue.

So it seems that there is and issue with the Coral USB and the ASM1042 USB 3.0 PCI card. The USB port pass through is not as fast as the PCI pass through but I really don't know what card to get, I have never had a device that was so picky about a USB PCI card.

hjonnala commented 1 year ago

can you please share the inference time results for USB port pass through vs USB PCI card pass through. Thanks!!

waynieack commented 1 year ago

To give some background on what I am doing, I am processing images from security camera footage to check for objects. When an event is recorded, the code selects a set of images and runs it through object detection. Currently it loads the model for each event and processes the images and exits, but I want to hold the model loaded (to reduce the processing time) and submit the images via an HTTP API and that is where the problem is because it crashes as we have already discussed above.

After running with the USB port passed through for a while with the model held loaded, it seemed to have just locked up at some point. I didn't get any logs for what happened at the time because there were way to many logs. So now I am back to the USB PCI card pass through loading the model on every run. I have seen it lock up when you interrupt an inference before its finished and it requires that you unplug the Coral and plug it back in (I have tried some USB reset script which didn't seem to work).

Both of these tests were done with the same group of images. The model is loaded before each set and inference is run on each image 1 time.

USB Port Passthrough: [perf: processor:tpu Coral TPU detection took: 33.12 ms] [perf: processor:tpu Coral TPU detection took: 33.97 ms] [perf: processor:tpu Coral TPU detection took: 33.59 ms] [perf: processor:tpu Coral TPU detection took: 32.74 ms] [perf: processor:tpu Coral TPU detection took: 32.63 ms] [perf: processor:tpu Coral TPU detection took: 33.22 ms] [perf: processor:tpu Coral TPU detection took: 34.02 ms] [perf: processor:tpu Coral TPU detection took: 36.11 ms] [perf: processor:tpu Coral TPU detection took: 32.67 ms] [perf: processor:tpu Coral TPU detection took: 33.74 ms] [perf: processor:tpu Coral TPU detection took: 33.59 ms] [perf: processor:tpu Coral TPU detection took: 33.50 ms] [perf: processor:tpu Coral TPU detection took: 34.33 ms] [perf: processor:tpu Coral TPU detection took: 34.20 ms]

USB PCI Card Passthrough: [perf: processor:tpu Coral TPU detection took: 21.93 ms] [perf: processor:tpu Coral TPU detection took: 21.76 ms] [perf: processor:tpu Coral TPU detection took: 22.14 ms] [perf: processor:tpu Coral TPU detection took: 22.38 ms] [perf: processor:tpu Coral TPU detection took: 21.06 ms] [perf: processor:tpu Coral TPU detection took: 21.33 ms] [perf: processor:tpu Coral TPU detection took: 21.58 ms] [perf: processor:tpu Coral TPU detection took: 22.15 ms] [perf: processor:tpu Coral TPU detection took: 20.96 ms] [perf: processor:tpu Coral TPU detection took: 20.90 ms] [perf: processor:tpu Coral TPU detection took: 21.58 ms] [perf: processor:tpu Coral TPU detection took: 23.53 ms] [perf: processor:tpu Coral TPU detection took: 21.64 ms] [perf: processor:tpu Coral TPU detection took: 21.19 ms]

Using the same model as above I also did the test with the examples/classify_image.py:

USB Port Passthrough: /coral/pycoral# python3 examples/classify_image.py --model /var/lib/zmeventnotification/models/coral_edgetpu/ssdlite_mobiledet_coco_qat_postprocess_edgetpu.tflite --labels /var/lib/zmeventnotification/models/coral_edgetpu/coco_indexed.names --input test_data/parrot.jpg ----INFERENCE TIME---- Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory. 43.6ms 27.5ms 28.5ms 29.5ms 27.7ms -------RESULTS-------- 207: 1.01903

USB PCI Card Passthrough: /coral/pycoral# python3 examples/classify_image.py --model /var/lib/zmeventnotification/models/coral_edgetpu/ssdlite_mobiledet_coco_qat_postprocess_edgetpu.tflite --labels /var/lib/zmeventnotification/models/coral_edgetpu/coco_indexed.names --input test_data/parrot.jpg Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory. 30.0ms 14.0ms 14.1ms 14.2ms 13.8ms -------RESULTS-------- 207: 1.01903