depthai_hand_tracking_unity_bridge crashes with the -xyz parameter

[novacto3]

Hello,

when I run depthai_hand_tracking_unity_bridge.py with the --xyz parameter, 9 out of 10 times, the process crashes.

The error is:

[1844301021AD6B1200] [2.1.2] [2.323] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'
F: [global] [         0] [ThreadN] dispatcherResponseServe:928  no request for this response: XLINK_WRITE_RESP 1

Traceback (most recent call last):
  File "C:\Škola\depthai-unity\unity_bridge\depthai_hand_tracking_unity_bridge.py", line 112, in <module>
    frame, hands, bag = tracker.next_frame()
                        ^^^^^^^^^^^^^^^^^^^^
  File "C:\Škola\depthai-unity\unity_bridge/depthai_hand_tracker\HandTracker.py", line 499, in next_frame
    in_video = self.q_video.get()

Any ideas how to fix this? Thanks!

[gespona]

Hi @novacto3 , seems the error you're reporting is happening in the external depthai_hand_tracker repository. I will check more in detail and come back here to you but meanwhile probably worth to check reported issues on the other repo

[gespona]

Hi @novacto3, I tried with fresh clone of the repo and --xyz option works fine all the time.

Could you check that you're using proper USB3 cable and USB3 port ? Sometimes issues arise if you don't use USB3

[novacto3]

Hi, @gespona! I am using USB 3 connected to a hub, which is connected to a USB C port. I can try it with a USB C cable straightaway later this week. However, when I tried it with different landmark models, it did not happen.

[gespona]

mmmmm I don't use any hub in the middle, but I tried with several combinations of params and worked fine all the time on my side. Error wise seems more comms issue tbh " no request for this response: XLINK_WRITE_RESP" Let me know if direct cable helps here

[gespona]

@novacto3 any luck with direct USB3 cable?

[novacto3]

I am so sorry! I had some health issues and then completely forgot to try it.

I confirm that it happens with direct USB3 cable, but also direct USB-C cable.

[gespona]

@novacto3 no worries ! Hope find you well now . Could you share the exact command line you're using to try reproduce on my side? Also could you share bit the HW specs ? It would be good if I can reproduce the error on my side 😅 Did you also tried to run the same command line with the original repo ? are you experiencing the same issue?

[novacto3]

@gespona

Hi!

The command I use is .\depthai_hand_tracking_unity_bridge.py -xyz (in the depthai-unity\unity_bridge folder). Crashes on both of the following HWs:

HW 1 (PC):

• OS: Windows 10
• CPU: Intel Core i5-9400F CPU @ 2.90GHz
• RAM: 64 GB
• GPU: NVIDIA GeForce RTX 2070

HW 2 (Lenovo laptop):

• OS: Windows 11
• CPU: Intel Core i7-1280P CPU @ 1.80GHz
• RAM: 32 GB
• GPU: Integrated Intel Iris Xe Graphics, 1 GB

I have not tried it in the original repo, because the -xyz parameter is only present in the unity hub (or I have not found it in the original repo).

[gespona]

Hi @novacto3 .. definetly we're not on the same page here ...

First just to mention that .\depthai_hand_tracking_unity_bridge.py is just a "wrapper" of repository depthai-hand-tracker

Bit more info just for curiosity: What OAK camera are you using? and which python version?

Could you execute following commands and copy paste output (if still crashing) ? git clone https://github.com/luxonis/depthai-unity.git cd .\depthai-unity\ git submodule update --init --recursive conda create -n hand-track-env python=3.10 conda activate hand-track-env cd .\unity_bridge\ python -m pip install -r requirements.txt python .\depthai_hand_tracking_unity_bridge.py -xyz

To test on original repo: cd .\depthai_hand_tracker\ python .\demo.py -xyz

[novacto3]

Hi, @gespona, yeah, I guess, sorry for that 😄

I am using Oak-D Lite with Python 3.12.

The first batch of commands gives me this error:

Listening...
Palm detection blob : C:\Škola\test\depthai-unity\unity_bridge\depthai_hand_tracker\models\palm_detection_sh4.blob
Landmark blob       : C:\Škola\test\depthai-unity\unity_bridge\depthai_hand_tracker\models\hand_landmark_lite_sh4.blob
[2024-06-12 21:45:37.903] [depthai] [warning] Skipping X_LINK_UNBOOTED device with name "1.3" ()
Internal camera FPS set to: 23
Sensor resolution: (1920, 1080)
Internal camera image size: 1152 x 648 - crop_w:0 pad_h: 252
896 anchors have been created
Creating pipeline...
Creating Color Camera...
Creating MonoCameras, Stereo and SpatialLocationCalculator nodes...
RGB calibration lens position: 0
Creating Palm Detection Neural Network...
Creating Hand Landmark Neural Network (2 threads)...
Pipeline created.
[1844301071DD7D1200] [1.3] [0.992] [NeuralNetwork(10)] [warning] Network compiled for 4 shaves, maximum available 12, compiling for 6 shaves likely will yield in better performance
Pipeline started - USB speed: SUPER
[1844301071DD7D1200] [1.3] [0.994] [NeuralNetwork(12)] [warning] Network compiled for 4 shaves, maximum available 12, compiling for 6 shaves likely will yield in better performance
[1844301071DD7D1200] [1.3] [1.008] [NeuralNetwork(10)] [warning] The issued warnings are orientative, based on optimal settings for a single network, if multiple networks are running in parallel the optimal settings may vary
[1844301071DD7D1200] [1.3] [1.009] [NeuralNetwork(12)] [warning] The issued warnings are orientative, based on optimal settings for a single network, if multiple networks are running in parallel the optimal settings may vary
[1844301071DD7D1200] [1.3] [2.174] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'
←[31mF: [global] [         0] [ThreadN] dispatcherResponseServe:928     no request for this response: XLINK_WRITE_RESP 1
←[0m
Traceback (most recent call last):
  File "C:\Škola\test\depthai-unity\unity_bridge\depthai_hand_tracking_unity_bridge.py", line 109, in <module>
    frame, hands, bag = tracker.next_frame()
  File "C:\Škola\test\depthai-unity\unity_bridge/depthai_hand_tracker\HandTracker.py", line 499, in next_frame
    in_video = self.q_video.get()
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'cam_out' (X_LINK_ERROR)'
[1844301071DD7D1200] [1.3] [1718221545.892] [host] [warning] Device crashed. Crash dump saved to C:\Users\Tomas\AppData\Local\Temp\1844301071DD7D1200-depthai_crash_dump.json

The second one this:

Palm detection blob : C:\Škola\test\depthai-unity\unity_bridge\depthai_hand_tracker\models\palm_detection_sh4.blob
Landmark blob       : C:\Škola\test\depthai-unity\unity_bridge\depthai_hand_tracker\models\hand_landmark_lite_sh4.blob
Internal camera FPS set to: 23
Sensor resolution: (1920, 1080)
Internal camera image size: 1152 x 648 - crop_w:0 pad_h: 252
896 anchors have been created
Creating pipeline...
Creating Color Camera...
Creating MonoCameras, Stereo and SpatialLocationCalculator nodes...
RGB calibration lens position: 0
Creating Palm Detection Neural Network...
Creating Hand Landmark Neural Network (2 threads)...
Pipeline created.
[1844301071DD7D1200] [1.3] [0.957] [NeuralNetwork(10)] [warning] Network compiled for 4 shaves, maximum available 12, compiling for 6 shaves likely will yield in better performance
[1844301071DD7D1200] [1.3] [0.960] [NeuralNetwork(12)] [Pipeline started - USB speed: SUPER
warning] Network compiled for 4 shaves, maximum available 12, compiling for 6 shaves likely will yield in better performance
[1844301071DD7D1200] [1.3] [0.974] [NeuralNetwork(10)] [warning] The issued warnings are orientative, based on optimal settings for a single network, if multiple networks are running in parallel the optimal settings may vary
[1844301071DD7D1200] [1.3] [0.974] [NeuralNetwork(12)] [warning] The issued warnings are orientative, based on optimal settings for a single network, if multiple networks are running in parallel the optimal settings may vary
[1844301071DD7D1200] [1.3] [2.143] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'
←[31mF: [global] [         0] [ThreadN] dispatcherResponseServe:928     no request for this response: XLINK_WRITE_RESP 1
←[0m
Traceback (most recent call last):
  File "C:\Škola\test\depthai-unity\unity_bridge\depthai_hand_tracker\demo.py", line 85, in <module>
    frame, hands, bag = tracker.next_frame()
  File "C:\Škola\test\depthai-unity\unity_bridge\depthai_hand_tracker\HandTracker.py", line 499, in next_frame
    in_video = self.q_video.get()
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'cam_out' (X_LINK_ERROR)'
[1844301071DD7D1200] [1.3] [1718221677.571] [host] [warning] Device crashed. Crash dump saved to C:\Users\Tomas\AppData\Local\Temp\1844301071DD7D1200-depthai_crash_dump.json

Crash dump here: 1844301071DD7D1200-depthai_crash_dump.json

[gespona]

@novacto3 Ok, that makes more sense ;) at least we have the same behaviour

Is that crash running on python 3.12 ? Could you try using python 3.10 ? It's very unlikely and I tried here with python 3.12 but unfortunatelly I don't have any OAK-D-lite at hand right now. Any chance you can test with other device ?

Could you confirm which version of depthai you're using ?

With that info I can ask internally if the issue is with the device

[gespona]

@novacto3 btw what is pretty curious, and just to confirm ... if you're not using -xyz params then the device is working fine right?

-xyz option is related to use depth .... could you confirm that depthai examples using depth are working for you?

[novacto3]

@gespona I tried it with Python 3.10, but it still has the same issue. Sadly, I only have Oak-D-Lite, but I have several of them, and it does not work with any of them. It works fine without the parameter. The DepthAI version is depthai-2.26.0.0.

BUT! It worked fine when I recompiled the hand-tracking models with six shaves (the depth works in the DepthAI viewer with sh4).

[gespona]

@novacto3 mmmmmm that's quite interesting and makes sense because according the warning we can see in the log, 4 shaves could not deliver optimal performance and in oak-d-lite could land on timing issues retrieveng the queues. Thanks for this feedback. Yesterday I reported internally. I will report that model with 6 shaves is working fine (-xyz) . All that said, if I understand correctly I will close the issue with next internal feedback.

Thanks for your patience

[gespona]

@novacto3 internal feedback: "It appears to be some timing/race issue, and it could be that changing the blob affects that."

Team is working on a fix, in case you need/want to use 4 shaves

[novacto3]

@gespona Thought so, thanks!

[gespona]

Fix is deployed on develop branch: https://github.com/luxonis/depthai-python/commit/aeecc04fee64236cec16c97a8f612b19e656d601