5 - onnxruntime_global_thread_pools_test (Failed)

[lvZic]

5: [ OK ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple3/4 (18 ms) 5: [----------] 15 tests from CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider (1852 ms total) 5: 5: [----------] Global test environment tear-down 5: [==========] 15 tests from 1 test suite ran. (1852 ms total) 5: [ PASSED ] 12 tests. 5: [ FAILED ] 3 tests, listed below: 5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple/1, where GetParam() = 1 5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple2/1, where GetParam() = 1 5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple3/1, where GetParam() = 1 5: 5: 3 FAILED TESTS 5/6 Test #5: onnxruntime_global_thread_pools_test ...***Failed 2.25 sec test 6 Start 6: onnxruntime_api_tests_without_env

6: Test command: /home/vic/proj/ARVR/onnxruntime/build/Linux/Release/onnxruntime_api_tests_without_env "--gtest_output=xml:/home/vic/proj/ARVR/onnxruntime/build/Linux/Release/onnxruntime_api_tests_without_env.Release.results.xml" 6: Test timeout computed to be: 3600 6: [==========] Running 1 test from 1 test suite. 6: [----------] Global test environment set-up. 6: [----------] 1 test from TestSessionOptions 6: [ RUN ] TestSessionOptions.SetIntraOpNumThreadsWithoutEnv 6: [ OK ] TestSessionOptions.SetIntraOpNumThreadsWithoutEnv (0 ms) 6: [----------] 1 test from TestSessionOptions (0 ms total) 6: 6: [----------] Global test environment tear-down 6: [==========] 1 test from 1 test suite ran. (0 ms total) 6: [ PASSED ] 1 test. 6/6 Test #6: onnxruntime_api_tests_without_env ...... Passed 0.05 sec

83% tests passed, 1 tests failed out of 6

Total Test time (real) = 66.75 sec

The following tests FAILED: 5 - onnxruntime_global_thread_pools_test (Failed) Errors while running CTest Output from these tests are in: /home/vic/proj/ARVR/onnxruntime/build/Linux/Release/Testing/Temporary/LastTest.log Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. Traceback (most recent call last): File "/home/vic/proj/ARVR/onnxruntime/tools/ci_build/build.py", line 1986, in sys.exit(main()) File "/home/vic/proj/ARVR/onnxruntime/tools/ci_build/build.py", line 1924, in main run_onnxruntime_tests(args, source_dir, ctest_path, build_dir, configs) File "/home/vic/proj/ARVR/onnxruntime/tools/ci_build/build.py", line 1346, in run_onnxruntime_tests run_subprocess(ctest_cmd, cwd=cwd, dll_path=dll_path) File "/home/vic/proj/ARVR/onnxruntime/tools/ci_build/build.py", line 528, in run_subprocess return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env) File "/home/vic/proj/ARVR/onnxruntime/tools/python/util/run.py", line 44, in run env=env, shell=shell) File "/usr/lib/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['/usr/bin/ctest', '--build-config', 'Release', '--verbose', '--timeout', '3600']' returned non-zero exit status 8.

[lvZic]

env: CUDA Version:11.1 libcudnn8_8.1.0.77-1+cuda11.2_amd64.deb
command: ./build.sh --build_shared_lib --config Release --use_cuda --cudnn_home /usr/local/cuda/ --cuda_home /usr/local/cuda

[fijipants]

I got the same issue on Windows 11, CUDA 11.5.2, cuDNN 8.3.3 (for CUDA 11.5).

Test #1 fails for me too:

1: C:\repo\onnxruntime\onnxruntime\test\providers\provider_test_utils.cc(251): error: The difference between expected[i] and output[i] is 0.0073043107986450195, which exceeds threshold, where
1: expected[i] evaluates to -1.9758384227752686,
1: output[i] evaluates to -1.9831427335739136, and
1: threshold evaluates to 0.004999999888241291.
1: i:1220, provider_type: CUDAExecutionProvider

Steps to reproduce:

• install vs 2019 or 2022
• check out latest or v1.11.0 or v1.10.0 or v1.8.0
• run the following:

.\build.bat --parallel 16 --config Release --build --update --use_cuda --cuda_home="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5" --cudnn_home="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5" --cuda_version 11.5
.\build.bat --parallel 16 --config Release --build_nodejs   --use_cuda --cuda_home="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5" --cudnn_home="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5" --cuda_version 11.5

NOTE: It works perfectly fine when I remove CUDA:

.\build.bat --parallel 16 --config Release --build --update
.\build.bat --parallel 16 --config Release --build_nodejs

[manickavela29]

facing the similar issue on building in windows , but getParams is initialized to 4.

5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple/1, where GetParam() = 4 5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple2/1, where GetParam() = 4 5: [ FAILED ] CApiTestGlobalThreadPoolsWithProviders/CApiTestGlobalThreadPoolsWithProvider.simple3/1, where GetParam() = 4

can anyone tell me what i should look out for or what is causing the issue?