search

luxonis / depthai-core

DepthAI C++ Library
MIT License
235 stars 127 forks source link

mutex violations in Device: causing race conditions and failures #79

Closed diablodale closed 3 years ago

diablodale commented 3 years ago

Hi. While working on issue https://github.com/luxonis/depthai-core/issues/71, I found errant code in Device.cpp

https://github.com/luxonis/depthai-core/blob/9f5d0861c07fea580c652f435c7e1422473ae079/src/device/Device.cpp#L401

Notice there is no variable name for the lock. Therefore the lock is created and instantly destroyed. It does not protect the scope. I have the fix (easy...just give the var a name) in my PR for issue #71. However, this fix then exposes failures elsewhere.

Setup

Repro

Do code review. See the variable has no name. Then correct the code like...

std::unique_lock<std::mutex> lock(this->rpcMutex);

and then run all the unit tests

Result

[proc] Executing command: ctest -j1 -C Debug -T test --output-on-failure
[ctest]    Site: DALE-XMG
[ctest]    Build name: Win32-ninja
[ctest] Test project C:/njs/depthai-core/build
[ctest]       Start  1: color_camera_node_test
[kit] OK running C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat amd64 10.0.19041.0, env vars: [["CL",""],["_CL_",""],["INCLUDE","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\ATLMFC\\include;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\include;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\ucrt;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\shared;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\um;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\winrt;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\cppwinrt"],["LIBPATH","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\ATLMFC\\lib\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\lib\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\lib\\x86\\store\\references;C:\\Program Files (x86)\\Windows Kits\\10\\UnionMetadata\\10.0.19041.0;C:\\Program Files (x86)\\Windows Kits\\10\\References\\10.0.19041.0;C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319"],["LINK",""],["_LINK_",""],["LIB","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\ATLMFC\\lib\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\lib\\x64;C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.19041.0\\ucrt\\x64;C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.19041.0\\um\\x64"],["PATH","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\\\Extensions\\Microsoft\\IntelliCode\\CLI;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\bin\\HostX64\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\VC\\VCPackages;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TestWindow;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TeamFoundation\\Team Explorer;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\MSBuild\\Current\\bin\\Roslyn;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Team Tools\\Performance Tools\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Team Tools\\Performance Tools;C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Common\\VSPerfCollectionTools\\vs2019\\\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Common\\VSPerfCollectionTools\\vs2019\\;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\Tools\\devinit;C:\\Program Files (x86)\\Windows Kits\\10\\bin\\10.0.19041.0\\x64;C:\\Program Files (x86)\\Windows Kits\\10\\bin\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\\\MSBuild\\Current\\Bin;C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\Tools\\;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Strawberry\\c\\bin;C:\\Strawberry\\perl\\site\\bin;C:\\Strawberry\\perl\\bin;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.1\\bin;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.1\\libnvvp;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2019.4.0\\;;C:\\Program Files\\Docker\\Docker\\resources\\bin;C:\\ProgramData\\DockerDesktop\\version-bin;C:\\Users\\dale\\AppData\\Local\\Microsoft\\WindowsApps;C:\\njs\\wslgit;;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\Llvm\\x64\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\CMake\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\Ninja;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\VC\\Linux\\bin\\ConnectionManagerExe"],["TMP","C:\\Users\\dale\\AppData\\Local\\Temp"],["FRAMEWORKDIR","C:\\Windows\\Microsoft.NET\\Framework64\\"],["FRAMEWORKDIR64","C:\\Windows\\Microsoft.NET\\Framework64\\"],["FRAMEWORKVERSION","v4.0.30319"],["FRAMEWORKVERSION64","v4.0.30319"],["UCRTCONTEXTROOT",""],["UCRTVERSION","10.0.19041.0"],["UNIVERSALCRTSDKDIR","C:\\Program Files (x86)\\Windows Kits\\10\\"],["VCINSTALLDIR","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\"],["VCTARGETSPATH",""],["WINDOWSLIBPATH","C:\\Program Files (x86)\\Windows Kits\\10\\UnionMetadata\\10.0.19041.0;C:\\Program Files (x86)\\Windows Kits\\10\\References\\10.0.19041.0"],["WINDOWSSDKDIR","C:\\Program Files (x86)\\Windows Kits\\10\\"],["WINDOWSSDKLIBVERSION","10.0.19041.0\\"],["WINDOWSSDKVERSION","10.0.19041.0\\"],["VISUALSTUDIOVERSION","16.0"]]
[ctest]  1/22 Test  #1: color_camera_node_test ..............   Passed    3.38 sec
[ctest]       Start  2: image_manip_node_test
[ctest]  2/22 Test  #2: image_manip_node_test ...............   Passed    3.29 sec
[ctest]       Start  3: pipeline_test
[ctest]  3/22 Test  #3: pipeline_test .......................   Passed    0.46 sec
[ctest]       Start  4: neural_network_test
[ctest]  4/22 Test  #4: neural_network_test .................   Passed    6.93 sec
[ctest]       Start  5: camera_preview
[ctest]  5/22 Test  #5: camera_preview ......................***Failed    2.94 sec
[ctest] statusfound -P
[ctest] -- arguments: C:/njs/depthai-core/build/examples/camera_preview.exe;
[ctest] -- After process executed,  produced the following exit code: 1
[ctest] CMake Error at C:/njs/depthai-core/examples/cmake/ExecuteTestTimeout.cmake:42 (message):
[ctest]    produced an error (1) while running
[ctest] 
[ctest] 
[ctest] 
[ctest]       Start  6: camera_video
[ctest]  6/22 Test  #6: camera_video ........................   Passed    5.05 sec
[ctest]       Start  7: mono_camera
[ctest]  7/22 Test  #7: mono_camera .........................   Passed    5.07 sec
[ctest]       Start  8: camera_mobilenet
[ctest]  8/22 Test  #8: camera_mobilenet ....................   Passed    5.06 sec
[ctest]       Start  9: webcam_mobilenet
[ctest]  9/22 Test  #9: webcam_mobilenet ....................   Passed    5.05 sec
[ctest]       Start 10: mjpeg_encoding
[ctest] 10/22 Test #10: mjpeg_encoding ......................***Failed    4.60 sec
[ctest] statusfound -P
[ctest] -- arguments: C:/njs/depthai-core/build/examples/mjpeg_encoding.exe;
[ctest] -- After process executed,  produced the following exit code: 3
[ctest] CMake Error at C:/njs/depthai-core/examples/cmake/ExecuteTestTimeout.cmake:42 (message):
[ctest]    produced an error (3) while running
[ctest] 
[ctest] 
[ctest] 
[ctest]       Start 11: h264_encoding
[ctest] 11/22 Test #11: h264_encoding .......................   Passed    5.05 sec
[ctest]       Start 12: stereo
[ctest] 12/22 Test #12: stereo ..............................   Passed    5.06 sec
[ctest]       Start 13: image_manip
[ctest] 13/22 Test #13: image_manip .........................   Passed    5.04 sec
[ctest]       Start 14: image_manip_warp
[ctest] 14/22 Test #14: image_manip_warp ....................   Passed    5.05 sec
[ctest]       Start 15: color_camera_control
[ctest] 15/22 Test #15: color_camera_control ................   Passed    5.05 sec
[ctest]       Start 16: system_information
[ctest] 16/22 Test #16: system_information ..................   Passed    5.04 sec
[ctest]       Start 17: device_queue_event
[ctest] 17/22 Test #17: device_queue_event ..................   Passed    5.06 sec
[ctest]       Start 18: opencv_support
[ctest] 18/22 Test #18: opencv_support ......................   Passed    5.06 sec
[ctest]       Start 19: mobilenet_device_side_decoding
[ctest] 19/22 Test #19: mobilenet_device_side_decoding ......   Passed    5.05 sec
[ctest]       Start 20: tiny_yolo_v3_device_side_decoding
[ctest] 20/22 Test #20: tiny_yolo_v3_device_side_decoding ...   Passed    5.05 sec
[ctest]       Start 21: tiny_yolo_v4_device_side_decoding
[ctest] 21/22 Test #21: tiny_yolo_v4_device_side_decoding ...   Passed    5.05 sec
[ctest]       Start 22: camera_mobilenet_sync
[ctest] 22/22 Test #22: camera_mobilenet_sync ...............   Passed    5.05 sec
[ctest] 
[ctest] 91% tests passed, 2 tests failed out of 22
[ctest] 
[ctest] Total Test time (real) = 102.53 sec
[ctest] 
[ctest] The following tests FAILED:
[ctest]       5 - camera_preview (Failed)
[ctest]      10 - mjpeg_encoding (Failed)
[ctest] Errors while running CTest
[ctest] CTest finished with return code 8
[extension] [3989] cmake.ctest finished (returned 8)

Expected

No failures.

Notes

With multiple test runs, I can get a more specific error. Naturally, since we are discussing mutex, this is a timing/race condition and the errors will likely vary. This one is very specific...

[ctest] 10/22 Test #10: mjpeg_encoding ......................***Failed    4.22 sec
[ctest] statusfound -P
[ctest] -- arguments: C:/njs/depthai-core/build/examples/mjpeg_encoding.exe;
[ctest] D:\a01\_work\9\s\src\vctools\crt\github\stl\src\mutex.cpp(64): mutex destroyed while busy
[ctest] -- After process executed,  produced the following exit code: Access violation
[ctest] CMake Error at C:/njs/depthai-core/examples/cmake/ExecuteTestTimeout.cmake:42 (message):
[ctest]    produced an error (Access violation) while running
[ctest]
diablodale commented 3 years ago

In another run camera_video still failed, but this time h264_encoding failed.

[proc] Executing command: ctest -j1 -C Debug -T test --output-on-failure
[ctest]    Site: DALE-XMG
[ctest]    Build name: Win32-ninja
[ctest] Test project C:/njs/depthai-core/build
[ctest]       Start  1: color_camera_node_test
[kit] OK running C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat amd64 10.0.19041.0, env vars: [["CL",""],["_CL_",""],["INCLUDE","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\ATLMFC\\include;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\include;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\ucrt;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\shared;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\um;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\winrt;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\cppwinrt"],["LIBPATH","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\ATLMFC\\lib\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\lib\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\lib\\x86\\store\\references;C:\\Program Files (x86)\\Windows Kits\\10\\UnionMetadata\\10.0.19041.0;C:\\Program Files (x86)\\Windows Kits\\10\\References\\10.0.19041.0;C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319"],["LINK",""],["_LINK_",""],["LIB","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\ATLMFC\\lib\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\lib\\x64;C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.19041.0\\ucrt\\x64;C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.19041.0\\um\\x64"],["PATH","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\\\Extensions\\Microsoft\\IntelliCode\\CLI;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\bin\\HostX64\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\VC\\VCPackages;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TestWindow;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TeamFoundation\\Team Explorer;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\MSBuild\\Current\\bin\\Roslyn;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Team Tools\\Performance Tools\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Team Tools\\Performance Tools;C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Common\\VSPerfCollectionTools\\vs2019\\\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Common\\VSPerfCollectionTools\\vs2019\\;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\Tools\\devinit;C:\\Program Files (x86)\\Windows Kits\\10\\bin\\10.0.19041.0\\x64;C:\\Program Files (x86)\\Windows Kits\\10\\bin\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\\\MSBuild\\Current\\Bin;C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\Tools\\;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Strawberry\\c\\bin;C:\\Strawberry\\perl\\site\\bin;C:\\Strawberry\\perl\\bin;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.1\\bin;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.1\\libnvvp;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2019.4.0\\;;C:\\Program Files\\Docker\\Docker\\resources\\bin;C:\\ProgramData\\DockerDesktop\\version-bin;C:\\Users\\dale\\AppData\\Local\\Microsoft\\WindowsApps;C:\\njs\\wslgit;;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\Llvm\\x64\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\CMake\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\Ninja;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\VC\\Linux\\bin\\ConnectionManagerExe"],["TMP","C:\\Users\\dale\\AppData\\Local\\Temp"],["FRAMEWORKDIR","C:\\Windows\\Microsoft.NET\\Framework64\\"],["FRAMEWORKDIR64","C:\\Windows\\Microsoft.NET\\Framework64\\"],["FRAMEWORKVERSION","v4.0.30319"],["FRAMEWORKVERSION64","v4.0.30319"],["UCRTCONTEXTROOT",""],["UCRTVERSION","10.0.19041.0"],["UNIVERSALCRTSDKDIR","C:\\Program Files (x86)\\Windows Kits\\10\\"],["VCINSTALLDIR","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\"],["VCTARGETSPATH",""],["WINDOWSLIBPATH","C:\\Program Files (x86)\\Windows Kits\\10\\UnionMetadata\\10.0.19041.0;C:\\Program Files (x86)\\Windows Kits\\10\\References\\10.0.19041.0"],["WINDOWSSDKDIR","C:\\Program Files (x86)\\Windows Kits\\10\\"],["WINDOWSSDKLIBVERSION","10.0.19041.0\\"],["WINDOWSSDKVERSION","10.0.19041.0\\"],["VISUALSTUDIOVERSION","16.0"]]
[ctest]  1/22 Test  #1: color_camera_node_test ..............   Passed    3.42 sec
[ctest]       Start  2: image_manip_node_test
[ctest]  2/22 Test  #2: image_manip_node_test ...............   Passed    3.94 sec
[ctest]       Start  3: pipeline_test
[ctest]  3/22 Test  #3: pipeline_test .......................   Passed    0.50 sec
[ctest]       Start  4: neural_network_test
[ctest]  4/22 Test  #4: neural_network_test .................   Passed    7.20 sec
[ctest]       Start  5: camera_preview
[ctest]  5/22 Test  #5: camera_preview ......................   Passed    5.07 sec
[ctest]       Start  6: camera_video
[ctest]  6/22 Test  #6: camera_video ........................***Failed    4.82 sec
[ctest] statusfound -P
[ctest] -- arguments: C:/njs/depthai-core/build/examples/camera_video.exe;
[ctest] -- After process executed,  produced the following exit code: 1
[ctest] CMake Error at C:/njs/depthai-core/examples/cmake/ExecuteTestTimeout.cmake:42 (message):
[ctest]    produced an error (1) while running
[ctest] 
[ctest] 
[ctest] 
[ctest]       Start  7: mono_camera
[ctest]  7/22 Test  #7: mono_camera .........................   Passed    5.06 sec
[ctest]       Start  8: camera_mobilenet
[ctest]  8/22 Test  #8: camera_mobilenet ....................   Passed    5.09 sec
[ctest]       Start  9: webcam_mobilenet
[ctest]  9/22 Test  #9: webcam_mobilenet ....................   Passed    5.06 sec
[ctest]       Start 10: mjpeg_encoding
[ctest] 10/22 Test #10: mjpeg_encoding ......................   Passed    5.05 sec
[ctest]       Start 11: h264_encoding
[ctest] 11/22 Test #11: h264_encoding .......................***Failed    3.67 sec
[ctest] statusfound -P
[ctest] -- arguments: C:/njs/depthai-core/build/examples/h264_encoding.exe;
[ctest] -- After process executed,  produced the following exit code: 1
[ctest] CMake Error at C:/njs/depthai-core/examples/cmake/ExecuteTestTimeout.cmake:42 (message):
[ctest]    produced an error (1) while running
[ctest] 
[ctest] 
[ctest] 
[ctest]       Start 12: stereo
[ctest] 12/22 Test #12: stereo ..............................   Passed    5.06 sec
[ctest]       Start 13: image_manip
[ctest] 13/22 Test #13: image_manip .........................   Passed    5.07 sec
[ctest]       Start 14: image_manip_warp
[ctest] 14/22 Test #14: image_manip_warp ....................   Passed    5.07 sec
[ctest]       Start 15: color_camera_control
[ctest] 15/22 Test #15: color_camera_control ................   Passed    5.06 sec
[ctest]       Start 16: system_information
[ctest] 16/22 Test #16: system_information ..................   Passed    5.06 sec
[ctest]       Start 17: device_queue_event
[ctest] 17/22 Test #17: device_queue_event ..................   Passed    5.06 sec
[ctest]       Start 18: opencv_support
[ctest] 18/22 Test #18: opencv_support ......................   Passed    5.08 sec
[ctest]       Start 19: mobilenet_device_side_decoding
[ctest] 19/22 Test #19: mobilenet_device_side_decoding ......   Passed    5.06 sec
[ctest]       Start 20: tiny_yolo_v3_device_side_decoding
[ctest] 20/22 Test #20: tiny_yolo_v3_device_side_decoding ...   Passed    5.06 sec
[ctest]       Start 21: tiny_yolo_v4_device_side_decoding
[ctest] 21/22 Test #21: tiny_yolo_v4_device_side_decoding ...   Passed    5.06 sec
[ctest]       Start 22: camera_mobilenet_sync
[ctest] 22/22 Test #22: camera_mobilenet_sync ...............   Passed    5.06 sec
[ctest] 
[ctest] 91% tests passed, 2 tests failed out of 22
[ctest] 
[ctest] Total Test time (real) = 104.67 sec
[ctest] 
[ctest] The following tests FAILED:
[ctest]       6 - camera_video (Failed)
[ctest]      11 - h264_encoding (Failed)
[ctest] Errors while running CTest
[ctest] CTest finished with return code 8
[extension] [5292] cmake.ctest finished (returned 8)
themarpe commented 3 years ago

Was playing around yesterday trying to reproduce. I could reproduce but wasn't able to do it in a debugger environment.

I tried getting WER working to produce a core dump on exception thrown, but couldn't get it to be saved.

If you are more familiar with Windows and have environment set up for cases like this and are able to capture a dump, will gladly take a closer look.

One idea that came to mind is also "termination" signal that is sent. CMake actually sends a KILL signal on execute_process TIMEOUT option. More specifically STOP then finds all derived processes (fork) and then sends KILL signal to all. Not sure about Windows procedure, but I presume its similar. Unfortunately it seems that sometimes it happens before timeout

Test #11: h264_encoding .......................***Failed    3.67 sec

So will have to investigate more. I also tried running for longer time (~5-10minutes) but never stumbled upon the error. Will have to do another session around this.

diablodale commented 3 years ago

I also see mutex issues before the "5 second" timeout as in OP and later post. Though, I can not confirm if the number of seconds reported is highly and always accurate. I think all that is managed by gtest -- I have little knowledge of gtest internals.

Searching around I found two related discussions. Perhaps they can help in your research https://stackoverflow.com/questions/22860087/mutex-destroyed-while-busy https://github.com/whoshuu/cpr/issues/46