search

luxonis / depthai-python

DepthAI Python Library
MIT License
348 stars 190 forks source link

OAK-1 MAX Log: 'Fatal error on MSS CPU: trap: 09, address: 802AD144' '0' #790

Open MAVProxyUser opened 1 year ago

MAVProxyUser commented 1 year ago

Unable to run YOLO ros example on the MAX-1?

[yolov4_spatial_node-1] [2023-03-18 21:54:48.006] [debug] Asset map dump: {"map":{"/node/1/__blob":{"alignment":64,"offset":0,"size":12172416}}}
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [ColorCamera(0)] [warning] LCM48 supported resolutions: 4_K / 4000X3000 / 5312X6000. Defaulting to 4_K
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(2)] [error] Camera not detected on socket: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(2)] [info] Using board socket: 1, id: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(3)] [error] Camera not detected on socket: 2
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.170] [MonoCamera(3)] [info] Using board socket: 2, id: 2
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.173] [system] [info] SpatialLocationCalculator shave buffer size '29696'B
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [2.458] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 09, address: 802AD144' '0'
[yolov4_spatial_node-1] [2023-03-18 21:54:51.522] [debug] DataOutputQueue (detections) closed
[yolov4_spatial_node-1] [2023-03-18 21:54:51.522] [debug] DataOutputQueue (depth) closed
[yolov4_spatial_node-1] [2023-03-18 21:54:51.523] [debug] Device about to be closed...
[yolov4_spatial_node-1] [2023-03-18 21:54:51.524] [debug] DataOutputQueue (preview) closed
[yolov4_spatial_node-1] [2023-03-18 21:54:51.525] [debug] Timesync thread exception caught: Couldn't read data from stream: '__timesync' (X_LINK_ERROR)
[yolov4_spatial_node-1] [2023-03-18 21:54:51.525] [debug] Log thread exception caught: Couldn't read data from stream: '__log' (X_LINK_ERROR)
[yolov4_spatial_node-1] [2023-03-18 21:54:51.718] [debug] Watchdog thread exception caught: Couldn't write data to stream: '__watchdog' (X_LINK_ERROR)
[yolov4_spatial_node-1] [2023-03-18 21:54:52.174] [debug] XLinkResetRemote of linkId: (0)
[yolov4_spatial_node-1] [2023-03-18 21:54:52.466] [debug] Device closed, 942
[yolov4_spatial_node-1] terminate called after throwing an instance of 'dai::XLinkReadError'
[yolov4_spatial_node-1]   what():  Couldn't read data from stream: '__rpc_main' (X_LINK_ERROR)
[ERROR] [yolov4_spatial_node-1]: process has died [pid 579125, exit code -6, cmd '/home/ubuntu/luxonis_oak_ws/install/depthai_examples/lib/depthai_examples/yolov4_spatial_node --ros-args --params-file /tmp/launch_params_yi0rp5n8 --params-file /tmp/launch_params_ofahoqpi --params-file /tmp/launch_params_nql3degb --params-file /tmp/launch_params_wng5cwr7 --params-file /tmp/launch_params_yi58p6s4 --params-file /tmp/launch_params_3bl7dpti'].
MAVProxyUser commented 1 year ago 
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.066] [system] [warning] PRINT:LeonCss: Calling IPC set assets
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.176] [system] [warning] PRINT:LeonCss: Asset: /node/1/__blob, pointer: 0x8abb5380, size: 12172416, alignment: 64
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.176] [ColorCamera(0)] [warning] LCM48 supported resolutions: 4_K / 4000X3000 / 5312X6000. Defaulting to 4_K
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.176] [MonoCamera(2)] [error] Camera not detected on socket: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.177] [MonoCamera(2)] [info] Using board socket: 1, id: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.177] [MonoCamera(3)] [error] Camera not detected on socket: 2
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.177] [MonoCamera(3)] [info] Using board socket: 2, id: 2
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.187] [system] [warning] PRINT:LeonMss: Calculating resources 
[yolov4_spatial_node-1] Number of cmx slices available 16 
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.181] [system] [info] SpatialLocationCalculator shave buffer size '29696'B
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.220] [system] [warning] PRINT:LeonMss: sippPalThreadCreate: Thread /SIPP created
[yolov4_spatial_node-1] Number of column of the first matrix should match with the number of rows of the second matrix 3x3 - 0
[yolov4_spatial_node-1] Unexpected trap ( 9) at address 0x802ad144
[yolov4_spatial_node-1] Bad trap, 0x9
[yolov4_spatial_node-1] PSR: 0xf34010c2 PC: 0x802ad148 NPC: 0x802ad14c TPC: 0x802ad144
[yolov4_spatial_node-1] G1:  0x00000000 G2: 0x00000000 G3:  0x00000000 G4:  0x00000000
[yolov4_spatial_node-1] G5:  0x00000000 G6: 0x806f43c0 G7:  0x00000000 I0:  0x00000000
[yolov4_spatial_node-1] I1:  0xaaaaa800 I2: 0x00000000 I3:  0x806f4390 I4:  0xaaaaaaab
[yolov4_spatial_node-1] I5:  0x82cc44c0 I6: 0x806f4328 I7:  0x8001d7e4 Y:   0x00000000
[yolov4_spatial_node-1] 
[yolov4_spatial_node-1] L0:  0x806f4504 L1: 0x82cc44c0 L2:  0x82cc44e4 L3:  0x806f4914
[yolov4_spatial_node-1] L4:  0x806f492c L5: 0x806f4944 L6:  0x806f4994 L7:  0x806f49a8
[yolov4_spatial_node-1] I0:  0x806f44f8 I1: 0x806f44e0 I2:  0x806f43c0 I3:  0x82cc44e4
[yolov4_spatial_node-1] I4:  0x806f439c I5: 0xaaaaaaab I6:  0x806f43c0 I7:  0x8014f3fc
[yolov4_spatial_node-1] SRA: 0x806f4374 SA0:0x806f4388 SA1: 0x806f4864 SA2: 0x00000000
[yolov4_spatial_node-1] SA3: 0x00000000 SA4:0x00000000 SA5: 0x806f4504
[yolov4_spatial_node-1] Unexpected trap ( 9) at address 0x802ad144
[yolov4_spatial_node-1] Bad trap, 0x9
[yolov4_spatial_node-1] PSR: 0xf34010c2 PC: 0x802ad148 NPC: 0x802ad14c TPC: 0x802ad144
[yolov4_spatial_node-1] G1:  0x00000000 G2: 0x00000000 G3:  0x00000000 G4:  0x00000000
[yolov4_spatial_node-1] G5:  0x00000000 G6: 0x806f43c0 G7:  0x00000000 I0:  0x00000000
[yolov4_spatial_node-1] I1:  0xaaaaa800 I2: 0x00000000 I3:  0x806f4390 I4:  0xaaaaaaab
[yolov4_spatial_node-1] I5:  0x82cc44c0 I6: 0x806f4328 I7:  0x8001d7e4 Y:   0x00000000
[yolov4_spatial_node-1] 
[yolov4_spatial_node-1] L0:  0x806f4504 L1: 0x82cc44c0 L2:  0x82cc44e4 L3:  0x806f4914
[yolov4_spatial_node-1] L4:  0x806f492c L5: 0x806f4944 L6:  0x806f4994 L7:  0x806f49a8
[yolov4_spatial_node-1] I0:  0x806f44f8 I1: 0x806f44e0 I2:  0x806f43c0 I3:  0x82cc44e4
[yolov4_spatial_node-1] I4:  0x806f439c I5: 0xaaaaaaab I6:  0x806f43c0 I7:  0x8014f3fc
[yolov4_spatial_node-1] SRA: 0x806f4374 SA0:0x806f4388 SA1: 0x806f4864 SA2: 0x00000000
[yolov4_spatial_node-1] SA3: 0x00000000 SA4:0x00000000 SA5: 0x806f4504
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.539] [system] [warning] PRINT:LeonCss: initial keepalive, countdown: 8
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [2.461] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 09, address: 802AD144' '0'
themarpe commented 1 year ago

@MAVProxyUser

Do you mind running crash_dump.py example afterwards & send that over? (WIP to have that done semi-automatically in the future)

MAVProxyUser commented 1 year ago

Oh @themarpe you mean this? https://github.com/luxonis/depthai-docs-website/commit/8497ef71df7475de50f2f5cb8a2339bc0cfbbe8c

Compiling develop branch per these notes now: Will check back. https://docs.luxonis.com/en/latest/pages/troubleshooting/

MAVProxyUser commented 1 year ago

@themarpe sorry brother... I can't get the develop branch to compile on my machine .

[2023-03-20 01:24:45.371] [debug] Library information - version: 2.20.2, commit: fc81e1b7d0c66813b9e75dbe83191d8c9e1db66e from 2023-03-19 20:09:09 +0100, build: 2023-03-20 04:46:23 +0000
[2023-03-20 01:24:45.377] [debug] Initialize - finished
[2023-03-20 01:24:45.655] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.24.tar.xz' open: 8ms, archive read: 273ms
Traceback (most recent call last):
  File "/usr/bin/stubgen", line 11, in <module>
    load_entry_point('mypy==0.761', 'console_scripts', 'stubgen')()
  File "mypy/stubgen.py", line 1559, in main
  File "mypy/stubgen.py", line 1452, in generate_stubs
  File "mypy/stubgenc.py", line 51, in generate_stub_for_c_module
  File "mypy/stubgenc.py", line 273, in generate_c_type_stub
  File "mypy/stubgenc.py", line 221, in generate_c_property_stub
TypeError: str object expected; got None
[2023-03-20 01:24:46.258] [debug] Resources - Archive 'depthai-device-fwp-cbab456e3481d680f7f125fbc3b5ec0435323a83.tar.xz' open: 8ms, archive read: 877ms
make[2]: *** [CMakeFiles/depthai.dir/build.make:987: depthai.cpython-38-aarch64-linux-gnu.so] Error 1
make[2]: *** Deleting file 'depthai.cpython-38-aarch64-linux-gnu.so'
make[1]: *** [CMakeFiles/Makefile2:187: CMakeFiles/depthai.dir/all] Error 2
MAVProxyUser commented 1 year ago

Ok I had to upgrade the mypy version 

 python3 -m pip install -U git+https://github.com/python/mypy.git
MAVProxyUser commented 1 year ago

ok @themarpe after some fussing around I figured out how to get the right branches. I triggered the crash... 

[yolov4_spatial_node-1] [1844301011A9460E00] [1.1] [1.129] [system] [warning] PRINT:LeonMss: sippPalThreadCreate: Thread /SIPP created
[yolov4_spatial_node-1] Number of column of the first matrix should match with the number of rows of the second matrix 3x3 - 0
[yolov4_spatial_node-1] Unexpected trap ( 9) at address 0x802936e8
[yolov4_spatial_node-1] Bad trap, 0x9
[yolov4_spatial_node-1] PSR: 0xf34010c2 PC: 0x802936ec NPC: 0x802936f0 TPC: 0x802936e8
[yolov4_spatial_node-1] G1:  0x00000000 G2: 0x00000000 G3:  0x00000000 G4:  0x00000000
[yolov4_spatial_node-1] G5:  0x00000000 G6: 0x806d7260 G7:  0x00000000 I0:  0x00000000
[yolov4_spatial_node-1] I1:  0xaaaaa800 I2: 0x00000000 I3:  0x806d7230 I4:  0xaaaaaaab
[yolov4_spatial_node-1] I5:  0x82ccce00 I6: 0x806d71c8 I7:  0x8001d568 Y:   0x00000000
[yolov4_spatial_node-1] 
[yolov4_spatial_node-1] L0:  0x806d73b4 L1: 0x82ccce00 L2:  0x82ccce24 L3:  0x806d77c4
[yolov4_spatial_node-1] L4:  0x806d77dc L5: 0x806d77f4 L6:  0x806d7844 L7:  0x806d7858
[yolov4_spatial_node-1] I0:  0x806d73a8 I1: 0x806d7390 I2:  0x806d7260 I3:  0x82ccce24
[yolov4_spatial_node-1] I4:  0x806d723c I5: 0xaaaaaaab I6:  0x806d7260 I7:  0x8014cf14
[yolov4_spatial_node-1] SRA: 0x806d7214 SA0:0x806d7228 SA1: 0x806d7714 SA2: 0x00000000
[yolov4_spatial_node-1] SA3: 0x00000000 SA4:0x00000000 SA5: 0x806d73b4
[yolov4_spatial_node-1] Unexpected trap ( 9) at address 0x802936e8
[yolov4_spatial_node-1] Bad trap, 0x9
[yolov4_spatial_node-1] PSR: 0xf34010c2 PC: 0x802936ec NPC: 0x802936f0 TPC: 0x802936e8
[yolov4_spatial_node-1] G1:  0x00000000 G2: 0x00000000 G3:  0x00000000 G4:  0x00000000
[yolov4_spatial_node-1] G5:  0x00000000 G6: 0x806d7260 G7:  0x00000000 I0:  0x00000000
[yolov4_spatial_node-1] I1:  0xaaaaa800 I2: 0x00000000 I3:  0x806d7230 I4:  0xaaaaaaab
[yolov4_spatial_node-1] I5:  0x82ccce00 I6: 0x806d71c8 I7:  0x8001d568 Y:   0x00000000
[yolov4_spatial_node-1] 
[yolov4_spatial_node-1] L0:  0x806d73b4 L1: 0x82ccce00 L2:  0x82ccce24 L3:  0x806d77c4
[yolov4_spatial_node-1] L4:  0x806d77dc L5: 0x806d77f4 L6:  0x806d7844 L7:  0x806d7858
[yolov4_spatial_node-1] I0:  0x806d73a8 I1: 0x806d7390 I2:  0x806d7260 I3:  0x82ccce24
[yolov4_spatial_node-1] I4:  0x806d723c I5: 0xaaaaaaab I6:  0x806d7260 I7:  0x8014cf14
[yolov4_spatial_node-1] SRA: 0x806d7214 SA0:0x806d7228 SA1: 0x806d7714 SA2: 0x00000000
[yolov4_spatial_node-1] SA3: 0x00000000 SA4:0x00000000 SA5: 0x806d73b4
[yolov4_spatial_node-1] [1844301011A9460E00] [1.1] [1.547] [system] [warning] PRINT:LeonCss: initial keepalive, countdown: 8
[yolov4_spatial_node-1] [1844301011A9460E00] [1.1] [2.401] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 09, address: 802936E8' '0'

but the crash_report script shows nothing. 

kfinisterre@dev1:~/depthai-python$ python3 ./examples/CrashReport/crash_report.py
There was no crash dump found on your device!
SzabolcsGergely commented 1 year ago

@MAVProxyUser the example doesn't work, because spatial examples require stereo cameras, and stereo depth. OAK-1-MAX has only 1 camera.

MAVProxyUser commented 1 year ago

@szabi-luxonis YOLO doesn't require stereo depth ... why should it here? It runs just fine via depthai_demo.py

This is literally off the same OAK-1-MAX camera. You can see there is no issue using YOLO with OAK-1-MAX, I'd have the expectation that the same detections would be returned by https://github.com/luxonis/depthai-ros/blob/humble/depthai_examples/launch/yolov4_publisher.launch.py

image

SzabolcsGergely commented 1 year ago

@MAVProxyUser you are running an example called: yolov4_spatial_node. Spatial node involves stereo depth. Error logs also point to missing stereo cameras, as expected.

[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(2)] [error] Camera not detected on socket: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(2)] [info] Using board socket: 1, id: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(3)] [error] Camera not detected on socket: 2
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.170] [MonoCamera(3)] [info] Using board socket: 2, id: 2
MAVProxyUser commented 1 year ago

@szabi-luxonis Is there a version of yolov4_publisher.launch.py that does not launch yolov4_spatial_node, or has a more intuitive flag to apply to the launch command-line so it starts with spatial disabled? There was seemingly nothing that indicated "stereo cameras" are "expected" until you pointed out that error buried in the logs. For me this is completely unexpected behavior for something that previously didn't indicate any need for a Stereo Cam. When I think YOLO, I don't at all think "depth is a requirement".

I had no clue it was dependent on spatial data until you just said so, and I subsequently looked at the code here: https://github.com/luxonis/depthai-ros/blob/humble/depthai_examples/launch/yolov4_publisher.launch.py#LL147C12-L147C19

 yolov4_spatial_node = launch_ros.actions.Node(
            package='depthai_examples', executable='yolov4_spatial_node',
            output='screen',
            parameters=[{'tf_prefix': tf_prefix},
                        {'camera_param_uri': camera_param_uri},
                        {'sync_nn': sync_nn},
                        {'nnName': nnName},
                        {'resourceBaseFolder': resourceBaseFolder},
                        {'monoResolution': monoResolution}])

As a new end user... can you help me out in getting a non-spatial example to run out of the box. I expected to be able to swap out my other Oak cameras in testing so I could compare performance. I have Oak-D-S2, Oak-D-Pro and two Oak-1 variants.

MAVProxyUser commented 1 year ago

At a glance @szabi-luxonis there is no non spatial YOLO example included, only this? https://github.com/luxonis/depthai-ros/blob/humble/depthai_examples/src/yolov4_spatial_publisher.cpp

SzabolcsGergely commented 1 year ago

Yes, that's true

MAVProxyUser commented 1 year ago

@szabi-luxonis do you have a suggestion as to how one would move forward getting an equivalent yolov4_publisher.launch.py that does not use spatial? Obviously several of your camera's are sold sans spatial options, and it would be nice to still have a YOLO example node for them.

SzabolcsGergely commented 1 year ago

@MAVProxyUser I would suggest opening a request on depthai-ros repository, and/or modifying the current example (you just need to remove extra code creating mono camera and stereo depth). Here is a reference in cpp which runs only yolo. Here is its equivalent in python. IF you don't need ROS, you can run these examples directly.

MAVProxyUser commented 1 year ago

ok @szabi-luxonis would you mind taking a look at this, and seeing if it seems sane? I tried to implement what you suggested above.

OAK-1-MAX non stereo / non spatial YOLO example node https://github.com/luxonis/depthai-ros/pull/262

I could use some assistance getting my Original OpenCV AI Kit: Oak-1 to work with this PR. For some reason it reports a blank "Device type:", so it can't be used with this example.

I get the following error:

[yolov4_image_node-1]   what():  There is no Camera data available corresponding to the the requested cameraId

Of course due to the blank response:

[component_container-1] [INFO] [1679549436.807773594] [oak]: Device type: