[BUG] Device crash using pipeline, which had no issues prior to this week

[tcbennun]

We've been using https://github.com/geaxgx/depthai_blazepose for some time without problems. This week, however, with the same hardware (OAK-1 and OAK-D Pro W), the camera crashes out as soon as a person enters the frame and a successful inference occurs (seemingly!)

We have this issue with any of the included models, whenever we run the model on-camera ("edge mode").

The program output is as follows.

❯ python demo.py -e
Pose detection blob file : /Users/tcb/repos/demos/depthai_blazepose/models/pose_detection_sh4.blob
Landmarks using blob file : /Users/tcb/repos/demos/depthai_blazepose/models/pose_landmark_full_sh4.blob
Internal camera FPS set to: 20
Internal camera image size: 1152 x 648 - pad_h: 252
Creating pipeline...
Creating Color Camera...
Creating Pose Detection pre processing image manip...
Creating Pose Detection Neural Network...
Creating Pose Detection post processing Neural Network...
Creating Landmark pre processing image manip...
Creating DiveideBy255 Neural Network...
Creating Landmark Neural Network...
Pipeline created.
[184430102103371300] [2.1] [1.964] [NeuralNetwork(5)] [warning] Network compiled for 1 shaves, maximum available 13, compiling for 6 shaves likely will yield in better performance
[184430102103371300] [2.1] [1.966] [NeuralNetwork(9)] [warning] Network compiled for 4 shaves, maximum available 13, compiling for 6 shaves likely will yield in better performance
[184430102103371300] [2.1] [2.198] [NeuralNetwork(4)] [warning] Network compiled for 4 shaves, maximum available 13, compiling for 6 shaves likely will yield in better performance
[184430102103371300] [2.1] [2.210] [NeuralNetwork(5)] [warning] The issued warnings are orientative, based on optimal settings for a single network, if multiple networks are running in parallel the optimal settings may vary
Pipeline started - USB speed: HIGH
[184430102103371300] [2.1] [2.210] [NeuralNetwork(9)] [warning] The issued warnings are orientative, based on optimal settings for a single network, if multiple networks are running in parallel the optimal settings may vary
[184430102103371300] [2.1] [2.210] [NeuralNetwork(4)] [warning] The issued warnings are orientative, based on optimal settings for a single network, if multiple networks are running in parallel the optimal settings may vary
Traceback (most recent call last):
  File "/Users/tcb/repos/demos/depthai_blazepose/demo.py", line 65, in <module>
    frame, body = tracker.next_frame()
                  ^^^^^^^^^^^^^^^^^^^^
  File "/Users/tcb/repos/demos/depthai_blazepose/BlazeposeDepthaiEdge.py", line 486, in next_frame
    res = marshal.loads(self.q_manager_out.get().getData())
                        ^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'manager_out' (X_LINK_ERROR)'
[2024-09-10 09:09:03.632] [depthai] [error] Device with id 184430102103371300 has crashed. Crash dump logs are stored in: /Users/tcb/repos/demos/depthai_blazepose/.cache/depthai/crashdumps/7f98df9fbd01d9480b59a14bda6e466b3458c828/crash_dump.json - please report to developers.

I was hoping someone could make something of the crash log:

crash_dump.json

Here's a pipeline graph;

We've tried various OAK devices, cables and PCs. We literally changed nothing since last week, when it was working! Is it possible there's a regression in the Luxonis firmware blob or something?

[tcbennun]

Note. We managed to fix the crash by downgrading the depthai package to 2.20.2, if that's a clue?

[moratom]

@jakaskerl do you mind checking if we can reproduce this and pinning down when the regression happens? Thanks.

[jakaskerl]

Seems to be from 2.20.2-->2.21.2