Closed diablodale closed 2 years ago
Here's the stack
ucrtbased.dll!00007ff971757475() (Unknown Source:0)
ucrtbased.dll!00007ff971757613() (Unknown Source:0)
ucrtbased.dll!00007ff97176d86d() (Unknown Source:0)
ucrtbased.dll!00007ff97176c886() (Unknown Source:0)
vcruntime140_1d.dll!00007ff9d1b0223a() (Unknown Source:0)
vcruntime140_1d.dll!00007ff9d1b02ec5() (Unknown Source:0)
vcruntime140_1d.dll!00007ff9d1b06ceb() (Unknown Source:0)
ntdll.dll!00007ff9dbb320cf() (Unknown Source:0)
ntdll.dll!00007ff9dbae1454() (Unknown Source:0)
ntdll.dll!00007ff9dbb30bfe() (Unknown Source:0)
KernelBase.dll!00007ff9d9384f69() (Unknown Source:0)
vcruntime140d.dll!00007ff9d1b1b460() (Unknown Source:0)
bootloader_version.exe!dai::XLinkStream::XLinkStream(const dai::XLinkConnection & conn, const std::string & name, unsigned __int64 maxWriteSize) Line 33 (c:\njs\depthai-core\src\xlink\XLinkStream.cpp:33)
bootloader_version.exe!dai::DeviceBootloader::init::__l2::<lambda>() Line 390 (c:\njs\depthai-core\src\device\DeviceBootloader.cpp:390)
bootloader_version.exe!std::invoke<void <lambda>(void)>(dai::DeviceBootloader::init::__l2::void <lambda>(void) && _Obj) Line 1586 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include\type_traits:1586)
bootloader_version.exe!std::thread::_Invoke<std::tuple<void <lambda>(void)>,0>(void * _RawVals) Line 56 (c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include\thread:56)
ucrtbased.dll!00007ff971774c7c() (Unknown Source:0)
kernel32.dll!00007ff9db657034() (Unknown Source:0)
ntdll.dll!00007ff9dbae2651() (Unknown Source:0)
Technically, it is from a debug build as follows:
"C:\Program Files\CMake\bin\cmake.exe" --build c:/njs/depthai-core/build --config Debug --target all -j 14 --
I suspect a relation between the unlimited_io_connection_test
and bootleader_version
. The majority of the time one or the other fails when they are run as part of a ctest
run.
Here is one of today's many fails, this time with...
env vars DEPTHAI_LEVEL=debug
, DEPTHAI_WATCHDOG=5000
\depthai-shared\include\depthai-shared\xlink\XLinkConstants.hpp
at top code changed to XLINK_WATCHDOG_TIMEOUT{5000}
ctest
...
[ctest] 6/61 Test #6: bootloader_config_test .............. Passed 0.93 sec
[ctest] Start 7: device_usbspeed_test
[ctest] 7/61 Test #7: device_usbspeed_test ................ Passed 15.99 sec
[ctest] Start 8: unlimited_io_connection_test
[ctest] 8/61 Test #8: unlimited_io_connection_test ........Exit code 0xc0000374
[ctest] ***Exception: 9.15 sec
[ctest] [2021-11-12 23:44:04.249] [debug] Library information - version: 2.11.1, commit: from , build: 2021-11-12 22:39:52 +0000
[ctest] [2021-11-12 23:44:04.254] [debug] Initialize - finished
[ctest] [2021-11-12 23:44:04.386] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.15.tar.xz' open: 5ms, archive read: 128ms
[ctest] [2021-11-12 23:44:04.854] [debug] Resources - Archive 'depthai-device-fwp-2021725697c9b570383b0597951a9d7b162e6182.tar.xz' open: 4ms, archive read: 599ms
[ctest] [2021-11-12 23:44:04.917] [debug] Device - OpenVINO version: 2021.4
[ctest] [2021-11-12 23:44:04.917] [debug] Using a custom watchdog value of 5000ms
[ctest] [14442C10C1A1C6D200] [5.023] [system] [info] Memory Usage - DDR: 0.12 / 359.07 MiB, CMX: 2.05 / 2.50 MiB, LeonOS Heap: 6.78 / 78.63 MiB, LeonRT Heap: 2.88 / 23.84 MiB
[ctest] [14442C10C1A1C6D200] [5.023] [system] [info] Temperatures - Average: 32.61 °C, CSS: 33.45 °C, MSS 31.77 °C, UPA: 32.73 °C, DSS: 32.49 °C
[ctest] [14442C10C1A1C6D200] [5.023] [system] [info] Cpu Usage - LeonOS 9.56%, LeonRT: 2.30%
[ctest] [2021-11-12 23:44:06.773] [debug] Schema dump: {"connections":[{"node1Id":0,"node1Output":"preview","node2Id":1,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":2,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":3,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":4,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":5,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":6,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":7,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":8,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":9,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":10,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":11,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":12,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":13,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":14,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":15,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":16,"node2Input":"in"}],"globalProperties":{"calibData":null,"cameraTuningBlobSize":null,"cameraTuningBlobUri":"","leonCssFrequencyHz":700000000.0,"leonMssFrequencyHz":700000000.0,"pipelineName":null,"pipelineVersion":null,"xlinkChunkSize":-1},"nodes":[[0,{"id":0,"ioInfo":{"inputConfig":{"blocking":false,"name":"inputConfig","queueSize":8,"type":3},"inputControl":{"blocking":true,"name":"inputControl","queueSize":8,"type":3},"isp":{"blocking":false,"name":"isp","queueSize":8,"type":0},"preview":{"blocking":false,"name":"preview","queueSize":8,"type":0},"raw":{"blocking":false,"name":"raw","queueSize":8,"type":0},"still":{"blocking":false,"name":"still","queueSize":8,"type":0},"video":{"blocking":false,"name":"video","queueSize":8,"type":0}},"name":"ColorCamera","properties":{"boardSocket":-1,"colorOrder":0,"fp16":false,"fps":30.0,"imageOrientation":-1,"initialControl":{"aeLockMode":false,"aeRegion":{"height":0,"priority":0,"width":0,"x":0,"y":0},"afRegion":{"height":0,"priority":0,"width":0,"x":0,"y":0},"antiBandingMode":0,"autoFocusMode":3,"awbLockMode":false,"awbMode":0,"brightness":0,"chromaDenoise":0,"cmdMask":0,"contrast":0,"effectMode":0,"expCompensation":0,"expManual":{"exposureTimeUs":0,"frameDurationUs":0,"sensitivityIso":0},"lensPosition":0,"lumaDenoise":0,"saturation":0,"sceneMode":0,"sharpness":0},"inputConfigSync":false,"interleaved":true,"ispScale":{"horizDenominator":0,"horizNumerator":0,"vertDenominator":0,"vertNumerator":0},"previewHeight":300,"previewKeepAspectRatio":true,"previewWidth":300,"resolution":0,"sensorCropX":-1.0,"sensorCropY":-1.0,"stillHeight":-1,"stillWidth":-1,"videoHeight":-1,"videoWidth":-1}}],[1,{"id":1,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out0"}}],[2,{"id":2,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out1"}}],[3,{"id":3,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out2"}}],[4,{"id":4,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out3"}}],[5,{"id":5,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out4"}}],[6,{"id":6,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out5"}}],[7,{"id":7,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out6"}}],[8,{"id":8,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out7"}}],[9,{"id":9,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out8"}}],[10,{"id":10,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out9"}}],[11,{"id":11,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out10"}}],[12,{"id":12,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out11"}}],[13,{"id":13,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out12"}}],[14,{"id":14,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out13"}}],[15,{"id":15,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out14"}}],[16,{"id":16,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out15"}}]]}
[ctest] [2021-11-12 23:44:06.773] [debug] Asset map dump: {"map":{}}
[ctest] [14442C10C1A1C6D200] [5.073] [system] [info] ImageManip internal buffer size '80640'B, shave buffer size '19456'B
[ctest] [14442C10C1A1C6D200] [5.073] [system] [info] SIPP (Signal Image Processing Pipeline) internal buffer size '122880'B
[ctest] [14442C10C1A1C6D200] [5.109] [system] [info] ColorCamera allocated resources: no shaves; cmx slices: [13-15]
[ctest] [14442C10C1A1C6D200] [5.109] [system] [info] ImageManip allocated resources: shaves: [15-15] no cmx slices.
[ctest] [14442C10C1A1C6D200] [6.024] [system] [info] Memory Usage - DDR: 41.27 / 359.07 MiB, CMX: 2.27 / 2.50 MiB, LeonOS Heap: 25.23 / 78.63 MiB, LeonRT Heap: 3.52 / 23.84 MiB
[ctest] [14442C10C1A1C6D200] [6.024] [system] [info] Temperatures - Average: 33.39 °C, CSS: 33.93 °C, MSS 32.97 °C, UPA: 33.21 °C, DSS: 33.45 °C
[ctest] [14442C10C1A1C6D200] [6.024] [system] [info] Cpu Usage - LeonOS 46.75%, LeonRT: 8.51%
[ctest] [14442C10C1A1C6D200] [7.025] [system] [info] Memory Usage - DDR: 41.27 / 359.07 MiB, CMX: 2.27 / 2.50 MiB, LeonOS Heap: 25.23 / 78.63 MiB, LeonRT Heap: 3.52 / 23.84 MiB
[ctest] [14442C10C1A1C6D200] [7.025] [system] [info] Temperatures - Average: 32.97 °C, CSS: 33.93 °C, MSS 32.73 °C, UPA: 32.01 °C, DSS: 33.21 °C
[ctest] [14442C10C1A1C6D200] [7.025] [system] [info] Cpu Usage - LeonOS 23.41%, LeonRT: 1.45%
[ctest] [14442C10C1A1C6D200] [8.026] [system] [info] Memory Usage - DDR: 41.27 / 359.07 MiB, CMX: 2.27 / 2.50 MiB, LeonOS Heap: 25.23 / 78.63 MiB, LeonRT Heap: 3.52 / 23.84 MiB
[ctest] [14442C10C1A1C6D200] [8.026] [system] [info] Temperatures - Average: 32.43 °C, CSS: 33.21 °C, MSS 32.97 °C, UPA: 31.29 °C, DSS: 32.25 °C
[ctest] [14442C10C1A1C6D200] [8.026] [system] [info] Cpu Usage - LeonOS 6.93%, LeonRT: 0.35%
[ctest] [14442C10C1A1C6D200] [9.027] [system] [info] Memory Usage - DDR: 41.27 / 359.07 MiB, CMX: 2.27 / 2.50 MiB, LeonOS Heap: 25.23 / 78.63 MiB, LeonRT Heap: 3.52 / 23.84 MiB
[ctest] [14442C10C1A1C6D200] [9.027] [system] [info] Temperatures - Average: 32.85 °C, CSS: 33.93 °C, MSS 32.49 °C, UPA: 32.49 °C, DSS: 32.49 °C
[ctest] [14442C10C1A1C6D200] [9.027] [system] [info] Cpu Usage - LeonOS 6.82%, LeonRT: 0.33%
[ctest] [14442C10C1A1C6D200] [10.028] [system] [info] Memory Usage - DDR: 41.27 / 359.07 MiB, CMX: 2.27 / 2.50 MiB, LeonOS Heap: 25.23 / 78.63 MiB, LeonRT Heap: 3.52 / 23.84 MiB
[ctest] [14442C10C1A1C6D200] [10.028] [system] [info] Temperatures - Average: 32.85 °C, CSS: 34.40 °C, MSS 32.97 °C, UPA: 31.53 °C, DSS: 32.49 °C
[ctest] [14442C10C1A1C6D200] [10.028] [system] [info] Cpu Usage - LeonOS 6.75%, LeonRT: 0.33%
[ctest] [2021-11-12 23:44:11.915] [debug] Device about to be closed...
[ctest] [2021-11-12 23:44:11.916] [debug] Timesync thread exception caught: Couldn't read data from stream: '__timesync' (X_LINK_ERROR)
[ctest] [2021-11-12 23:44:11.919] [debug] Log thread exception caught: Couldn't read data from stream: '__log' (X_LINK_ERROR)
[ctest]
[ctest] Start 9: bootloader_version
[ctest] 9/61 Test #9: bootloader_version .................. Passed 2.24 sec
...
if I cd to build/examples
and manually run bootloader_version.exe
, it often works...but not always as you can see in my OP.
If I run tests/unlimited_io_connection_test.exe
, I can see watchdog failures. Are these failures creating an environment where later tests fail?
C:\njs\depthai-core\build\tests>unlimited_io_connection_test.exe
[2021-11-12 23:59:15.781] [debug] Library information - version: 2.11.1, commit: from , build: 2021-11-12 22:39:52 +0000
[2021-11-12 23:59:15.786] [debug] Initialize - finished
[2021-11-12 23:59:15.908] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.15.tar.xz' open: 5ms, archive read: 118ms
[2021-11-12 23:59:16.343] [debug] Resources - Archive 'depthai-device-fwp-2021725697c9b570383b0597951a9d7b162e6182.tar.xz' open: 5ms, archive
read: 555ms
[2021-11-12 23:59:16.441] [debug] Device - OpenVINO version: 2021.4
[2021-11-12 23:59:16.442] [debug] Using a custom watchdog value of 5000ms
[14442C10C1A1C6D200] [261.221] [system] [info] Memory Usage - DDR: 0.12 / 359.07 MiB, CMX: 2.05 / 2.50 MiB, LeonOS Heap: 6.78 / 78.63 MiB, LeonRT Heap: 2.88 / 23.84 MiB
[14442C10C1A1C6D200] [261.221] [system] [info] Temperatures - Average: 31.65 ┬░C, CSS: 32.01 ┬░C, MSS 31.53 ┬░C, UPA: 31.29 ┬░C, DSS: 31.77 ┬░C
[14442C10C1A1C6D200] [261.221] [system] [info] Cpu Usage - LeonOS 9.52%, LeonRT: 2.29%
[2021-11-12 23:59:18.250] [debug] Schema dump: {"connections":[{"node1Id":0,"node1Output":"preview","node2Id":1,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":2,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":3,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":4,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":5,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":6,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":7,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":8,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":9,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":10,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":11,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":12,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":13,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":14,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":15,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":16,"node2Input":"in"}],"globalProperties":{"calibData":null,"cameraTuningBlobSize":null,"cameraTuningBlobUri":"","leonCssFrequencyHz":700000000.0,"leonMssFrequencyHz":700000000.0,"pipelineName":null,"pipelineVersion":null,"xlinkChunkSize":-1},"nodes":[[0,{"id":0,"ioInfo":{"inputConfig":{"blocking":false,"name":"inputConfig","queueSize":8,"type":3},"inputControl":{"blocking":true,"name":"inputControl","queueSize":8,"type":3},"isp":{"blocking":false,"name":"isp","queueSize":8,"type":0},"preview":{"blocking":false,"name":"preview","queueSize":8,"type":0},"raw":{"blocking":false,"name":"raw","queueSize":8,"type":0},"still":{"blocking":false,"name":"still","queueSize":8,"type":0},"video":{"blocking":false,"name":"video","queueSize":8,"type":0}},"name":"ColorCamera","properties":{"boardSocket":-1,"colorOrder":0,"fp16":false,"fps":30.0,"imageOrientation":-1,"initialControl":{"aeLockMode":false,"aeRegion":{"height":0,"priority":0,"width":0,"x":0,"y":0},"afRegion":{"height":0,"priority":0,"width":0,"x":0,"y":0},"antiBandingMode":0,"autoFocusMode":3,"awbLockMode":false,"awbMode":0,"brightness":0,"chromaDenoise":0,"cmdMask":0,"contrast":0,"effectMode":0,"expCompensation":0,"expManual":{"exposureTimeUs":0,"frameDurationUs":0,"sensitivityIso":0},"lensPosition":0,"lumaDenoise":0,"saturation":0,"sceneMode":0,"sharpness":0},"inputConfigSync":false,"interleaved":true,"ispScale":{"horizDenominator":0,"horizNumerator":0,"vertDenominator":0,"vertNumerator":0},"previewHeight":300,"previewKeepAspectRatio":true,"previewWidth":300,"resolution":0,"sensorCropX":-1.0,"sensorCropY":-1.0,"stillHeight":-1,"stillWidth":-1,"videoHeight":-1,"videoWidth":-1}}],[1,{"id":1,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out0"}}],[2,{"id":2,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out1"}}],[3,{"id":3,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out2"}}],[4,{"id":4,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out3"}}],[5,{"id":5,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out4"}}],[6,{"id":6,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out5"}}],[7,{"id":7,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out6"}}],[8,{"id":8,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out7"}}],[9,{"id":9,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out8"}}],[10,{"id":10,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out9"}}],[11,{"id":11,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out10"}}],[12,{"id":12,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out11"}}],[13,{"id":13,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out12"}}],[14,{"id":14,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out13"}}],[15,{"id":15,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out14"}}],[16,{"id":16,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out15"}}]]}
[2021-11-12 23:59:18.253] [debug] Asset map dump: {"map":{}}
[14442C10C1A1C6D200] [261.274] [system] [info] ImageManip internal buffer size '80640'B, shave buffer size '19456'B
[14442C10C1A1C6D200] [261.274] [system] [info] SIPP (Signal Image Processing Pipeline) internal buffer size '122880'B
[14442C10C1A1C6D200] [261.310] [system] [info] ColorCamera allocated resources: no shaves; cmx slices: [13-15]
[14442C10C1A1C6D200] [261.310] [system] [info] ImageManip allocated resources: shaves: [15-15] no cmx slices.
[14442C10C1A1C6D200] [262.222] [system] [info] Memory Usage - DDR: 41.27 / 359.07 MiB, CMX: 2.27 / 2.50 MiB, LeonOS Heap: 25.23 / 78.63 MiB, LeonRT Heap: 3.52 / 23.84 MiB
[14442C10C1A1C6D200] [262.222] [system] [info] Temperatures - Average: 32.79 ┬░C, CSS: 34.16 ┬░C, MSS 32.97 ┬░C, UPA: 31.53 ┬░C, DSS: 32.49 ┬░C
[14442C10C1A1C6D200] [262.222] [system] [info] Cpu Usage - LeonOS 44.67%, LeonRT: 8.31%
[14442C10C1A1C6D200] [263.223] [system] [info] Memory Usage - DDR: 41.27 / 359.07 MiB, CMX: 2.27 / 2.50 MiB, LeonOS Heap: 25.23 / 78.63 MiB, LeonRT Heap: 3.52 / 23.84 MiB
[14442C10C1A1C6D200] [263.223] [system] [info] Temperatures - Average: 32.13 ┬░C, CSS: 33.93 ┬░C, MSS 31.53 ┬░C, UPA: 31.53 ┬░C, DSS: 31.53 ┬░C
[14442C10C1A1C6D200] [263.223] [system] [info] Cpu Usage - LeonOS 26.13%, LeonRT: 1.60%
[14442C10C1A1C6D200] [264.224] [system] [info] Memory Usage - DDR: 41.27 / 359.07 MiB, CMX: 2.27 / 2.50 MiB, LeonOS Heap: 25.23 / 78.63 MiB, LeonRT Heap: 3.52 / 23.84 MiB
[14442C10C1A1C6D200] [264.224] [system] [info] Temperatures - Average: 32.31 ┬░C, CSS: 32.97 ┬░C, MSS 32.49 ┬░C, UPA: 31.77 ┬░C, DSS: 32.01 ┬░C
[14442C10C1A1C6D200] [264.224] [system] [info] Cpu Usage - LeonOS 6.90%, LeonRT: 0.35%
[14442C10C1A1C6D200] [265.225] [system] [info] Memory Usage - DDR: 41.27 / 359.07 MiB, CMX: 2.27 / 2.50 MiB, LeonOS Heap: 25.23 / 78.63 MiB, LeonRT Heap: 3.52 / 23.84 MiB
[14442C10C1A1C6D200] [265.225] [system] [info] Temperatures - Average: 32.13 ┬░C, CSS: 33.69 ┬░C, MSS 31.77 ┬░C, UPA: 31.29 ┬░C, DSS: 31.77 ┬░C
[14442C10C1A1C6D200] [265.225] [system] [info] Cpu Usage - LeonOS 6.77%, LeonRT: 0.33%
[14442C10C1A1C6D200] [266.226] [system] [info] Memory Usage - DDR: 41.27 / 359.07 MiB, CMX: 2.27 / 2.50 MiB, LeonOS Heap: 25.23 / 78.63 MiB, LeonRT Heap: 3.52 / 23.84 MiB
[14442C10C1A1C6D200] [266.226] [system] [info] Temperatures - Average: 32.49 ┬░C, CSS: 33.45 ┬░C, MSS 32.25 ┬░C, UPA: 31.77 ┬░C, DSS: 32.49 ┬░C
[14442C10C1A1C6D200] [266.226] [system] [info] Cpu Usage - LeonOS 6.73%, LeonRT: 0.34%
numFrames: 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
[2021-11-12 23:59:23.399] [debug] Device about to be closed...
[2021-11-12 23:59:23.517] [debug] Timesync thread exception caught: Couldn't read data from stream: '__timesync' (X_LINK_ERROR)
[2021-11-12 23:59:23.517] [debug] Log thread exception caught: Couldn't read data from stream: '__log' (X_LINK_ERROR)
[2021-11-12 23:59:25.513] [debug] XLinkResetRemote of linkId: (0)
[2021-11-12 23:59:25.763] [debug] Device closed, 2363
[2021-11-12 23:59:26.434] [debug] Device - OpenVINO version: 2021.4
[2021-11-12 23:59:26.436] [debug] Using a custom watchdog value of 5000ms
[14442C10C1A1C6D200] [3.746] [system] [info] Memory Usage - DDR: 0.12 / 359.07 MiB, CMX: 2.05 / 2.50 MiB, LeonOS Heap: 6.78 / 78.63 MiB, LeonRT Heap: 2.88 / 23.84 MiB
[14442C10C1A1C6D200] [3.746] [system] [info] Temperatures - Average: 31.77 ┬░C, CSS: 33.45 ┬░C, MSS 30.56 ┬░C, UPA: 31.04 ┬░C, DSS: 32.01 ┬░C
[14442C10C1A1C6D200] [3.746] [system] [info] Cpu Usage - LeonOS 10.04%, LeonRT: 2.43%
[2021-11-12 23:59:28.210] [debug] Schema dump: {"connections":[{"node1Id":0,"node1Output":"preview","node2Id":1,"node2Input":"inputImage"},{"node1Id":1,"node1Output":"out","node2Id":2,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":3,"node2Input":"inputImage"},{"node1Id":3,"node1Output":"out","node2Id":4,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":5,"node2Input":"inputImage"},{"node1Id":5,"node1Output":"out","node2Id":6,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":7,"node2Input":"inputImage"},{"node1Id":7,"node1Output":"out","node2Id":8,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":9,"node2Input":"inputImage"},{"node1Id":9,"node1Output":"out","node2Id":10,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":11,"node2Input":"inputImage"},{"node1Id":11,"node1Output":"out","node2Id":12,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":13,"node2Input":"inputImage"},{"node1Id":13,"node1Output":"out","node2Id":14,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":15,"node2Input":"inputImage"},{"node1Id":15,"node1Output":"out","node2Id":16,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":17,"node2Input":"inputImage"},{"node1Id":17,"node1Output":"out","node2Id":18,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":19,"node2Input":"inputImage"},{"node1Id":19,"node1Output":"out","node2Id":20,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":21,"node2Input":"inputImage"},{"node1Id":21,"node1Output":"out","node2Id":22,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":23,"node2Input":"inputImage"},{"node1Id":23,"node1Output":"out","node2Id":24,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":25,"node2Input":"inputImage"},{"node1Id":25,"node1Output":"out","node2Id":26,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":27,"node2Input":"inputImage"},{"node1Id":27,"node1Output":"out","node2Id":28,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":29,"node2Input":"inputImage"},{"node1Id":29,"node1Output":"out","node2Id":30,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":31,"node2Input":"inputImage"},{"node1Id":31,"node1Output":"out","node2Id":32,"node2Input":"in"}],"globalProperties":{"calibData":null,"cameraTuningBlobSize":null,"cameraTuningBlobUri":"","leonCssFrequencyHz":700000000.0,"leonMssFrequencyHz":700000000.0,"pipelineName":null,"pipelineVersion":null,"xlinkChunkSize":-1},"nodes":[[0,{"id":0,"ioInfo":{"inputConfig":{"blocking":false,"name":"inputConfig","queueSize":8,"type":3},"inputControl":{"blocking":true,"name":"inputControl","queueSize":8,"type":3},"isp":{"blocking":false,"name":"isp","queueSize":8,"type":0},"preview":{"blocking":false,"name":"preview","queueSize":8,"type":0},"raw":{"blocking":false,"name":"raw","queueSize":8,"type":0},"still":{"blocking":false,"name":"still","queueSize":8,"type":0},"video":{"blocking":false,"name":"video","queueSize":8,"type":0}},"name":"ColorCamera","properties":{"boardSocket":-1,"colorOrder":0,"fp16":false,"fps":30.0,"imageOrientation":-1,"initialControl":{"aeLockMode":false,"aeRegion":{"height":0,"priority":0,"width":0,"x":0,"y":0},"afRegion":{"height":0,"priority":0,"width":0,"x":0,"y":0},"antiBandingMode":0,"autoFocusMode":3,"awbLockMode":false,"awbMode":0,"brightness":0,"chromaDenoise":0,"cmdMask":0,"contrast":0,"effectMode":0,"expCompensation":0,"expManual":{"exposureTimeUs":0,"frameDurationUs":0,"sensitivityIso":0},"lensPosition":0,"lumaDenoise":0,"saturation":0,"sceneMode":0,"sharpness":0},"inputConfigSync":false,"interleaved":false,"ispScale":{"horizDenominator":0,"horizNumerator":0,"vertDenominator":0,"vertNumerator":0},"previewHeight":300,"previewKeepAspectRatio":true,"previewWidth":300,"resolution":0,"sensorCropX":-1.0,"sensorCropY":-1.0,"stillHeight":-1,"stillWidth":-1,"videoHeight":-1,"videoWidth":-1}}],[1,{"id":1,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[2,{"id":2,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out0"}}],[3,{"id":3,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[4,{"id":4,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out1"}}],[5,{"id":5,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[6,{"id":6,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out2"}}],[7,{"id":7,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[8,{"id":8,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out3"}}],[9,{"id":9,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[10,{"id":10,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out4"}}],[11,{"id":11,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[12,{"id":12,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out5"}}],[13,{"id":13,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[14,{"id":14,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out6"}}],[15,{"id":15,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[16,{"id":16,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out7"}}],[17,{"id":17,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[18,{"id":18,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out8"}}],[19,{"id":19,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[20,{"id":20,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out9"}}],[21,{"id":21,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[22,{"id":22,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out10"}}],[23,{"id":23,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[24,{"id":24,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out11"}}],[25,{"id":25,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[26,{"id":26,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out12"}}],[27,{"id":27,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[28,{"id":28,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out13"}}],[29,{"id":29,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[30,{"id":30,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out14"}}],[31,{"id":31,"ioInfo":{"inputConfig":{"blocking":true,"name":"inputConfig","queueSize":8,"type":3},"inputImage":{"blocking":true,"name":"inputImage","queueSize":8,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"ImageManip","properties":{"initialConfig":{"cropConfig":{"cropRatio":1.0,"cropRect":{"xmax":0.0,"xmin":0.0,"ymax":0.0,"ymin":0.0},"cropRotatedRect":{"angle":0.0,"center":{"x":0.0,"y":0.0},"size":{"height":0.0,"width":0.0}},"enableCenterCropRectangle":false,"enableRotatedRect":false,"normalizedCoords":true,"widthHeightAspectRatio":1.0},"enableCrop":false,"enableFormat":false,"enableResize":false,"formatConfig":{"flipHorizontal":false,"type":7},"resizeConfig":{"bgBlue":0,"bgGreen":0,"bgRed":0,"enableRotation":false,"enableWarp4pt":false,"enableWarpMatrix":false,"height":0,"keepAspectRatio":true,"lockAspectRatioFill":false,"normalizedCoords":true,"rotationAngleDeg":0.0,"warpBorderReplicate":false,"warpFourPoints":[],"warpMatrix3x3":[],"width":0},"reusePreviousImage":false,"skipCurrentImage":false},"inputConfigSync":false,"numFramesPool":4,"outputFrameSize":1048576}}],[32,{"id":32,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"out15"}}]]}
[2021-11-12 23:59:28.222] [debug] Asset map dump: {"map":{}}
[14442C10C1A1C6D200] [3.922] [system] [info] ImageManip internal buffer size '203904'B, shave buffer size '22528'B
[14442C10C1A1C6D200] [3.922] [system] [info] SIPP (Signal Image Processing Pipeline) internal buffer size '122880'B
[14442C10C1A1C6D200] [3.958] [system] [info] ColorCamera allocated resources: no shaves; cmx slices: [13-15]
[14442C10C1A1C6D200] [3.958] [system] [info] ImageManip allocated resources: shaves: [15-15] no cmx slices.
numFrames: 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
[2021-11-12 23:59:28.872] [debug] Device about to be closed...
[2021-11-12 23:59:28.989] [debug] Log thread exception caught: Couldn't read data from stream: '__log' (X_LINK_ERROR)
[2021-11-12 23:59:28.990] [debug] Timesync thread exception caught: Couldn't read data from stream: '__timesync' (X_LINK_ERROR)
[2021-11-12 23:59:30.696] [debug] Watchdog thread exception caught: Couldn't write data to stream: '__watchdog' (X_LINK_ERROR)
[2021-11-12 23:59:31.007] [debug] XLinkResetRemote of linkId: (0)
[2021-11-12 23:59:31.013] [debug] Device closed, 2140
===============================================================================
All tests passed (2 assertions in 2 test cases)
I've seen a pattern that is visible when DEPTHAI_LEVEL=debug
.
Tests that usually successful have two Resources - Archive
and then Device - OpenVINO
Tests that fail have the same two Resources - Archive
but crash and therefore have no Device - OpenVINO
Looking at the code and behavior, I smell a race condition bug related to getLazyTarXzFunction
and the mtxDevice
mutex.
Usually successful
[ctest] 10: Test command: "C:\Program Files\CMake\bin\cmake.exe" "-DTIMEOUT_SECONDS=20" "-P" "C:/njs/depthai-core/examples/cmake/ExecuteTestTimeout.cmake" "C:/njs/depthai-core/build/examples/rgb_camera_control.exe"
[ctest] 10: Environment variables:
[ctest] 10: UBSAN_OPTIONS=halt_on_error=1
[ctest] 10: Test timeout computed to be: 1500
[ctest] 10: -- found -P
[ctest] 10: -- arguments: C:/njs/depthai-core/build/examples/rgb_camera_control.exe;
[ctest] 10: [2021-11-13 00:57:44.988] [debug] Library information - version: 2.11.1, commit: from , build: 2021-11-12 23:55:37 +0000
[ctest] 10: [2021-11-13 00:57:44.992] [debug] Initialize - finished
[ctest] 10: [2021-11-13 00:57:45.103] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.15.tar.xz' open: 5ms, archive read: 108ms
[ctest] 10: [2021-11-13 00:57:45.534] [debug] Resources - Archive 'depthai-device-fwp-2021725697c9b570383b0597951a9d7b162e6182.tar.xz' open: 4ms, archive read: 540ms
[ctest] 10: [2021-11-13 00:57:45.657] [debug] Device - OpenVINO version: 2021.4
[ctest] 10: [2021-11-13 00:57:45.657] [debug] Using a custom watchdog value of 5000ms
[ctest] 10: [14442C10C1A1C6D200] [6.395] [system] [info] Memory Usage - DDR: 0.12 / 359.07 MiB, CMX: 2.05 / 2.50 MiB, LeonOS Heap: 6.78 / 78.63 MiB, LeonRT Heap: 2.88 / 23.84 MiB
...
Fail
[ctest] 48: Test command: "C:\Program Files\CMake\bin\cmake.exe" "-DTIMEOUT_SECONDS=20" "-P" "C:/njs/depthai-core/examples/cmake/ExecuteTestTimeout.cmake" "C:/njs/depthai-core/build/examples/spatial_tiny_yolo_v4.exe"
[ctest] 48: Environment variables:
[ctest] 48: UBSAN_OPTIONS=halt_on_error=1
[ctest] 48: Test timeout computed to be: 1500
[ctest] 48: -- found -P
[ctest] 48: -- arguments: C:/njs/depthai-core/build/examples/spatial_tiny_yolo_v4.exe;
[ctest] 48: [2021-11-13 01:10:03.183] [debug] Library information - version: 2.11.1, commit: from , build: 2021-11-12 23:55:37 +0000
[ctest] 48: [2021-11-13 01:10:03.187] [debug] Initialize - finished
[ctest] 48: [2021-11-13 01:10:03.357] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.15.tar.xz' open: 5ms, archive read: 166ms
[ctest] 48: [2021-11-13 01:10:03.833] [debug] Resources - Archive 'depthai-device-fwp-2021725697c9b570383b0597951a9d7b162e6182.tar.xz' open: 4ms, archive read: 644ms
[ctest] 48: -- After process executed, produced the following exit code: -2147483645
[ctest] 48: CMake Error at C:/njs/depthai-core/examples/cmake/ExecuteTestTimeout.cmake:42 (message):
[ctest] 48: produced an error (-2147483645) while running
[ctest] 48:
[ctest] 48:
[ctest] 48/61 Test #48: spatial_tiny_yolo_v4 ................***Failed 13.54 sec
I've got a good repo that so far always fails. As in the OP it is src\xlink\XLinkStream.cpp line 33 At that throw, in teh debugger, I see the following
thread::join()
from DeviceBootleader::close()
line 439 which was called from the destructor ~DeviceBootleader()
.dai::XLinkStream::XLinkStream(const dai::XLinkConnection & conn, const std::string & name, unsigned __int64 maxWriteSize)
which was called from DeviceBootloader::init::__l2::<lambda>() Line 392
which is "prepare watchdog thread" My question is... the first part of that prepare watchdog thread which is constructing an XLinkStream and needs a connection
. But on the other thread in the DeviceBootleader::close()
is it blocked in the join()
...and look 3 lines above that join
and I see a connection->close() and =nullptr. Are these the same connection
? Is there a race condition where that connection is being closed by one thread while the constructor on the other thread is trying to work with that connection?
I put a spdlog::debug on line 25 to output spdlog::debug("connlinkid= {} streamId= {}", conn.getLinkId(), streamId);
Here is what I got. See...the first loop has a linkid 0 but while the loop is running/sleeping it turns into -1.
Found device with name: 14442C10C1A1C6D200-ma2480
[2021-11-13 21:30:43.436] [debug] connlinkid= 0 streamId= 0
Version: 0.0.15
[2021-11-13 21:30:43.438] [debug] DeviceBootloader about to be closed...
[2021-11-13 21:30:43.439] [debug] connlinkid= 0 streamId= 3735936685
[2021-11-13 21:30:43.491] [debug] connlinkid= -1 streamId= 3735936685
[2021-11-13 21:30:43.550] [debug] connlinkid= -1 streamId= 3735936685
[2021-11-13 21:30:43.603] [debug] connlinkid= -1 streamId= 3735936685
[2021-11-13 21:30:43.668] [debug] connlinkid= -1 streamId= 3735936685
Exception thrown at 0x00007FFA73854F69 in bootloader_version.exe: Microsoft C++ exception: std::runtime_error at memory location 0x000000E18FFFF820.
I think this is one source of the crashes. I see
First issue...you dereference and loose the nature of a shared_pointer. The XLinkStream()
will not share ownership of it. So when this raw connection
by reference is passed to the XLinkStream
constructor...and in another thread the destructor/close() closes and then sets the shared_ptr to nullptr...that set to nullptr probably releases the last owner of the shared_ptr
I see 14 places where this unsafe behavior of passing a raw reference from a shared_ptrXLinkStream[^:]*\(
and also look for dereferencing connection "connection" but be careful as in some places it is "this->connection".
Second issue...There is also that the ~DeviceBootloader()
calls close()
, which calls connection->close(). That then sets deviceLinkId = -1
which we can see in the loop debug above. RAII with connection will call close itself. It you resolve ownership issues, then the closing can automatically happen.
I grabbed ownership using auto hack = connection;
in many places and now I can get further in running bootloader_version.exe
. Now EXE sometimes exits good=0x0. And sometimes it exits 0x1 due to dai::XLinkReadError
being thrown in XLinkStream::readRaw()
being called by DeviceBootloader::init
Line 417 "// Dummy read (wait till link falls down)"
Found device with name: 14442C10C1A1C6D200-ma2480
[2021-11-13 23:00:23.124] [debug] connlinkid= 0 streamId= 0
Version: 0.0.15
[2021-11-13 23:00:23.125] [debug] DeviceBootloader about to be closed...
[2021-11-13 23:00:23.126] [debug] connlinkid= 0 streamId= 1
The program '[18084] bootloader_version.exe' has exited with code 1 (0x1).
I think i've found a bug in XLink usage. After hack above of holding ownership, I kept having the app exit with exitcode 1. I couldn't find anywhere an exception or windows SEH. So I looked into XLink itself.
XLink has several places where it exit()
. Naturally, no library should ever exit(). Big nono. This is probably what is causing the app to exit(1) through some cascade when DeviceBootloader::init()
at the very end of itself spawns the separate watchdog thread and calls stream.write(reset);
.
After enabling XLink logging(warn) I can get a 90% reliable app exit 1 and it appears like this
[2021-11-15 15:27:59.358] [debug] Library information - version: 2.11.1, commit: 928f5ead870b59e772baeeff14bea68b8d1d9f96 from 2021-11-12 22:14:49 +0100, build: 2021-11-15 13:22:30 +0000
[2021-11-15 15:27:59.921] [debug] Resources - Archive 'depthai-device-fwp-2021725697c9b570383b0597951a9d7b162e6182.tar.xz' open: 4ms, archive read: 554ms
[2021-11-15 15:28:00.037] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.15.tar.xz' open: 3ms, archive read: 109ms
[2021-11-15 15:28:00.040] [debug] Initialize - finished
Found device with name: 14442C10C1A1C6D200-ma2480
[2021-11-15 15:28:01.197] [debug] connlinkid= 0 streamId= 0
creating... init::watchdogThread
Version: 0.0.15
DeviceBootloader about to be closed...
::close before join()
[2021-11-15 15:28:01.199] [debug] connlinkid= 0 streamId= 1
init::watchdogThread 2 before loop
init::watchdogThread 2 after loop 1
init::watchdogThread 2 after readRaw
init::watchdogThread 2 before sleep in try
init::watchdogThread 2 after sleep in try
init::watchdogThread 2 exiting thread lambda...
F: [global] [ 0] [EventRead00Thr] dispatcherEventReceive:91 Duplicate id detected.
::close after join()
The program '[19484] bootloader_version.exe' has exited with code 1 (0x1).
The logs above are explaining that the watchdog lambda finished (doesn't always), the join() in close()
completed, but the final code of close()
didn't run and the app didn't exit from main. Instead something else (XLink?) exited the app with (1).
Found the XLink issue. I've logged it separately in your xlink repo https://github.com/luxonis/XLink/issues/18
Also found use of XLinkConnection without ownership two times in DataOutputQueue
and DataInputQueue
In general, a dev should never pass a shared_ptr as const& because you always (if it is a shared_ptr) have the refcount incremented. So a fix in these two areas is to have a private member of the classes a std::shared_ptr<XLinkConnection>
and then pass it in const and in initializer std::move it. E.g.
DataInputQueue::DataInputQueue(const std::shared_ptr<XLinkConnection> conn, const std::string& streamName, unsigned int maxSize, bool blocking)
: connection(std::move(conn)), queue(maxSize, blocking), name(streamName) {
// open stream with default XLINK_USB_BUFFER_MAX_SIZE write size
XLinkStream stream(*connection, name, device::XLINK_USB_BUFFER_MAX_SIZE);
Thanks for digging into this issue.
XLink is a library provided by Intel for communication between device and host, doesn't have the best error handling (using exit
as you pointed out), we didn't get into rewriting it, or isolating issues at deinitialization.
Looking forward to the pull request.
The majority of these thread/ownership code issues are in depthai (not xlink). These are not xlink bugs.
These bugs point to core design/architectural issues in depthai. Depthai classes need to have new members to hold ownership of things on which they operate, a cascade of ownerships related to them (definitely connection, maybe device, etc) to populate those members, destructors that will cleanup, etc.
Your team should have conversations and whiteboard on how you want to manage ownership using the examples and repro cases and code I've pointed to above. This is your code, you have the corpus of knowledge. I don't have the corpus, don't have architectural ownership, and don't have the role to edict how you architect. This is work your team needs to do.
My hacks are private as I'm not overhauling your classes...just quick grabbing ownership so I can start actually coding my app. Its messy, still not entirely thread safe, and definitely not for merging for wide usage.
The xlink issue is separate from all the thread/ownership issues and the xlink "bug" is separate at https://github.com/luxonis/XLink/issues/18. Those xlink changes are isolated and trivial. I can make a PR for those xlink issues.
I agree, we will look into these once we finish current efforts.
When you are ready, I can push my hacks somewhere so you can see how ownership is missing and how I've got temporary hacks in place for some of the most problematic. There is definitely a better approach by shuffling classes/members as my changes are temporary hacks, repetitive, and still have some race conditions.
Main reason for this "design" was because of current XLink limitations, Data queues cannot unblock the XLink R/W calls, etc... The connection is passed as a const ref as its only needed for the duration of the call, to create XLinkStream "wrapper".
This is also the main pain point which needs work on - both from XLink and core perspective - to make more resilient to device connect/disconnects.
That said, the causes where this shows are mostly related to trying to connect to a device which isn't in a "ready" state or going out of a connection in a not clean way. In your testing, do you repeatedly call eg. bootloader_version? How does it behave if you run the example with some time in between or a device power cycle? Does the error rate remain the same? Or does it work in that case?
Will read through the rest of the observations and dig into XLink more soon
...but since depthai is multithreaded there is no safe "duration of the call". One must always get ownership using the intention of shared_ptr....one must pass it by value so that shared_ptr will do the refcount++ and therefore the connection will live during the duration of the call.
My repro is consistantly to run your ctest as per https://github.com/luxonis/depthai-core#running-tests while I have ensured that all are ON -> DEPTHAI_BUILD_TESTS, DEPTHAI_BUILD_EXAMPLES, and DEPTHAI_TEST_EXAMPLES. The pain points tend to be tests 7-12. When they run in their natural sequence, depthai fails the majority of the time. 😟 Even after all my hack fixes, I still get occassional failures (though I have some ideas of race conditions that might be causing them). To date, I have never seen a 100% pass rate on your full test suite.
From what I can see in your CI https://github.com/luxonis/depthai-core/runs/3735947677?check_suite_focus=true you only run two tests. You never run your full test suite. I suspect you are missing lots of crashes. I know some of your full suite needs a gui...and the test timeout dance. But tests 1-9 need no UI you should be able to CI them easily. Overall, I encourage you to run (at minimum) tests 1-11 (rgb_preview) in your CI.
One of my views...is if a dependency's (e.g. depthai) doesn't pass its own test suite with a clean build, then there is a low chance of my project using said dependency to be successful. I'll always be chasing issues/bugs due to underlying dependency issues. "Can't build on quicksand." That's why I started digging into depthai in Spring and returned as planned now in Fall and am hardcore trying to get depthai's test suite to pass.
My particular usage of depthai will be a DLL that "plugs" into a 3rd party EXE. I can't crash that EXE nor can I expect a user to powercycle OAK on every config change because the user rapidly makes changes...probably hundreds of times in a day. I will integrate with a UI that allows the user to dynamically at runtime choosing one or more OAK devices, choose the config for that particulate OAK (resolution, rgb, mono, depth, etc) all runtime without app restarts or crashes. Its not tenable for depthai to crash when a user stops depthai, chooses a different rgb resolution, and then restarts depthai. I won't be able to recommend OAK hardware nor sell my solution supporting it.
At the last commit of https://github.com/diablodale/depthai-core/commits/hack_257_ownership are all the hacks for threads/ownership. You'll also see some commits before that which I'll push as formal PRs later this week or next to fix other issues I've already opened.
And a PR is still forthcoming on XLink itself. That one is easy to fix.
...but since depthai is multithreaded there is no safe "duration of the call". One must always get ownership using the intention of shared_ptr....one must pass it by value so that shared_ptr will do the refcount++ and therefore the connection will live during the duration of the call.
It does, the caller doesn't decrement its own refcount for the duration till it returns from that function, so as long as the caller has a valid ref, the called function will have it as well.
From what I can see in your CI https://github.com/luxonis/depthai-core/runs/3735947677?check_suite_focus=true you only run two tests. You never run your full test suite. I suspect you are missing lots of crashes. I know some of your full suite needs a gui...and the test timeout dance. But tests 1-9 need no UI you should be able to CI them easily. Overall, I encourage you to run (at minimum) tests 1-11 (rgb_preview) in your CI.
We've yet to setup local runners with HW to test against and be tied to CI. We do however test these locally currently (unscalable, working on creating runners - security is a bit of an issue with GH self-hosted runners)
I can't crash that EXE nor can I expect a user to powercycle OAK on every config change because the user rapidly makes changes...probably hundreds of times in a day.
I understand - I agree this not being a solution, was just a question to see what could be under the crash, as some of this behavior we've seen before. Makes it easier to start exploring the root cause. Still, how does running that example in a said manner manifest itself? Same behavior as with running in quick succession?
The reason I'm asking is that there are some XLink issues which are the root of how to resolve this "completely correctly", and it gives a quick glance on the severity of jumping to them.
And thanks for the snippets and the XLink PR - will take into consideration when circling back to core.
Please don't take my pointing of errant things in a bad way. My intention is to point to the bug, not people. We're human, make mistakes, fix them, and move on. 👍
I already debugged and know the source of the first wave of crashes.
One of the best test cases is bootloader_version
. Beautifully simple. ✨ It exposes multiple depthai thread/ownership issues + the xlink exit() issue.
Build it Windows x64 Debug. Repeatedly run the EXE in a batch loop. Over and over. You'll be lucky if you can do 2 runs.
Bugs include spawning "watchdog" threads but pass const& shared_ptr
After you fix the ownership of the connection then the code will run longer to then suddenly terminate due to the exit() issue in XLink. Once I found it, its an obvious bug and easy/isolated to fix. I have one question on it I'll put in the PR notes. I'll open that PR within an hour.
A class diagram would be useful so I can see the hierarchy and cross-usage. Does such a diagram exist? That can help me independently -or- if we have a interactive convo assists on which classes need to hold ownership to others. Given time, I'll grok it myself as I debug/fix. I would rather code my app, but I need to stablize depthai-core enough for my use cases.
I would rather code my app, but I need to stablize depthai-core enough for my use cases.
Sorry for the disappointment @diablodale . And thanks for the help.
good news 🎉 I've got a 100% test suite passing tests 1->50. This is a dramatic improvement!
test 51 rgb_depth_aligned.exe
fails with [StereoDepth(3)] [error] RGB camera calibration missing, aligning to RGB won't work
which likely a completely separate issue. I vaguely remember something about early units not having cali in firmware...but I didn't think I was in that batch. https://docs.luxonis.com/en/latest/pages/calibration/
I'll remove all my debugging logging and push the changes I made. It's a much better approach than my earlier hacks. Can still be improved but its a great start.
AWESOME thanks @diablodale !
OK. I've pushed my fixes/workarounds to https://github.com/diablodale/depthai-core/tree/hack_257_ownership I learned/surfaced a lot...
All the above issues are true. I won't repeat them again here.
I was able two days ago to get 100% pass rate on the full 61 item test suite. My approach didn't call connection::close()
and instead waited for RAII to close it. But test 2 image_manip_node_test
took ~70 seconds to fully shutdown. It was likely blocking on a semaphore in XLink (probably from the dai::DataQueue) and after that time finally unblocked due to a timeout or sporadic wakeup. Can't wait 70 seconds. ⏱ I changed to call connection::close()
and had to deal with a chain of problems.
The depthai codebase uses the close()
to cascade a series of fails/errors that leads to a harsh shutdown and unblocking everything. But...depthai is multithreaded and our CPUs running threads concurrently. So you have to think about multiple threads doing different things all at the same time.
Thread 1 - the watchdog 🐕🦺 Watchdog was spawned to be in a tight loop that is reading and writing to an XLinkStream across an XLinkConnection. The same boring conversation of "are you there", "yes I'm here", "are you there", "yes I'm here", ...
Thread 2 - the DataOutputQueue 🏭
The DataOutputQueue thread was spawned to be in a loop to read from an XLinkStream across an XLinkConnection. When the device sends data from device->host it arrives here. On arrival, the thread calls parseMessageToADatatype()
to transform the raw XLink packet into something useful for depthai.
Thread 3 - the destroyer ☠ This thread might be the main thread. Something that is running directed tasks from the app. "Get an image". "Change resolution." "Shutdown."
All three threads are running in parallel. None of the threads are synchronized between themselves.
Imagine the main app wants to shutdown the current OAK session. Thread 3☠ initiates that work and with "shutdown" it eventually calls connection->close()
. This is the close() written above. Further, the original depthai code also set connection = nullptr
which is even more problematic because then both the connection is invalid and the shared_ptr<connection>
itself is empty. See my branch for the fix to not invalidate the shared_ptr itself.
Thread 1🐕🦺 and thread 2🏭 need a valid connection to run correctly. Therefore they throw errors and exceptions which all need to be managed. See my branch for fixes. But... threads 1 and 2 are not at some beautiful/easy location in their loop. They are in random locations of code in their loop and therefore the XLink connection is closed at random times.
Thread 2🏭 is problematic. First, it is the thread that often blocks leading the current codebase to do the harsh connection::close()
. And... it is running in parallel of yet another thread...
Thread 4 - XLink 🔗 This thread is internal to the XLink library. It takes commands from an internal event queue and acts on them. It creates packets of data to send from device->host and stores them in a circular queue. It also receives commands and acts on them. One such command is to shutdown/die. On shutdown, it releases the memory of all the packets currently in the circular queue and then sets all the values in that queue to zero/null.
Now we have 3 threads in a death scenario.
parseMessageToADatatype(packet*)
and that param is a pointer to one of the entries in the circular buffer being managed by thread 4🔗💥Hard crash of SEGFAULT and memory violations.
Thread 2 was actively using that packet*
. Calling readIntLE(packet->data)
, json::from_msgpack(packet->data)
, std::vector(packet->data)
, etc.
The packet pointer (as a 64-bit uint) is valid. But the struct memory location to which it pointed was set to zero by thread 4🔗.
And...the struct to which it pointed also has a data
pointer. The memory to which data
pointed was deallocated by thread 4🔗. So depending on timing, the packet struct might have zeros in it. Or...it might have what looks like good values but the memory to which the values points has been deallocated. 🤹♀️
These hard crashes due to memory violations are VERY difficult to manage. These must be fixed. Since these are "hardware" errors, they are outside standard C++ exception handling mechanisms. Linux has SIG handlers but the C++ object unwinding breaks down. Windows has proprietary SEH __try
but it is difficult to manage alongside standard C++ try
.
Windows+MSVC has a compiler option that is a workaround. /EHa
enables many SEH (memory violation type errors) to be caught using the C++ standard clause catch(...)
. This is my workaround you can see in the branch.
It is Windows+MSVC only.
It requires ALL compilation units to be compiled with /EHa
. That means all the Hunter components, depthai-core, depthai-shared, etc.... everything that links into the same DLL or LIB must all be /EHa
. https://docs.microsoft.com/en-us/cpp/build/reference/eh-exception-handling-model?view=msvc-170#structured-and-standard-c-exception-handling
More analysis. All above issues in depthai and XLink continue. Focusing on the XLink code...
XLinkDispatcherImpl.c
has problems in dispatcherCloseLink()
around https://github.com/luxonis/XLink/blob/1eaa307a3f0b5c390ee10c23daaf5fa45aa974e5/src/shared/XLinkDispatcherImpl.c#L464
XLink_sem_wait()
and XLink_sem_post()
before operating on the stream.getPacketFromStream()
a packet stored in slot 2. And then immediately calls releasePacketFromStream(). Notice it has no pointers or slots identifying which packet to release. 😬 releasePacketFromStream()
blindly releases/destroys the packet at the "front". So it destroys the packet in slot 0 (instead of the packet it just get
from slot 2. Crash💥 That slot 0 packet is being used by the app. XLinkStreamReset()
blindly memset(stream, 0)
without checking for packets still being used. If the above is fixed, I think this will be safe.I'm attempting to fix the XLink issues without introducing C++ into XLink. 🙄 Yet, it is so much easier with C++ RAII and unique/shared_ptr...but does change the interface and cascade some work. If you are working on this in parallel, let's share ideas. If you are not yet, then I'll continue to push forward as I need to get this fixed so my app meets my reliability standards.
Thanks you! To my knowledge we are not working on this in parallel but @themarpe can confirm.
I finished the thread+ownership fixes I've planned for this wave. Tests above that often failed in fast loops -- passed. Full 61 test suite (6 iterations) in a loop -- passed
Overnight I'll run rgb_depth_aligned.exe
to see if there are any noticable memory leaks.
If all is good after the run, I'll first make a PR for the XLink changes. I want to get some more eyes on it for style+approach. The needed depthai-changes are dependent on these XLink changes. I'll make a PR for those after we have consensus on XLink.
Thank you!
Great news @diablodale
Regarding bringing C++ into XLink, there are initial attempts over at device_search_improvements
which also brings in libusb as a crossplatform lib (including Windows).
It however only touches the USB part, and doesn't refactor anything else. I think attacking parts of the codebase that way will in time turn it into C++, while reducing a lot of unnecessary complexity.
A fix for this overall issue is in https://github.com/diablodale/depthai-core/commit/3ebfd7dc17217e31e6ebda21b5bdac05cee450e0 It first requires XLink move semantics of https://github.com/luxonis/XLink/pull/29
Mistakenly closed this issue...
@diablodale the move semantic is now merged.
Will you open the PR against the core
with above mentioned fixes? Will work great in direction of both this issue and starting on zerocopy
@themarpe, yes. Will do before end of year. I'll rebase it on to current develop
branch.
Using the
ctest
instructions in this repo, or manually running/build/examples/bootloader_version.exe
...leads to a hard crash maybe 50% of the time. If I run that EXE in vscode to debug it, I can seeXLinkStream::XLinkStream(const XLinkConnection& conn, const std::string& name, std::size_t maxWriteSize) : streamName(name)
constructor throws and nothing is catching so it hard crashes.Setup
Repro
build
thenctest
Result
Expected
No errors or crashes. All tests to pass.
Notes
verbose ctest
Debugger tracing
Run and debug
/build/examples/bootloader_version.exe
directly in vscode so I can debug the crashthe crash is in
src\xlink\XLinkStream.cpp
line 33https://github.com/luxonis/depthai-core/blob/c1b697afa432c8d620b097ba57e20a0ffd2ec52d/src/xlink/XLinkStream.cpp#L32-L34
streamID does equal INVALID_STREAM_ID. It is
0xDEADDEAD
. and this->streamname = __watchdog