Closed franz closed 6 months ago
Can you please also post all the relevant system info? Seems like the same tests as on Aurora (except they timeout on Aurora). Can you also post here the results for RC1?
1.1-RC1 results:
99% tests passed, 1 tests failed out of 948
Label Time Summary:
cuda = 66.70 sec*proc (26 tests)
internal = 115.76 sec*proc (73 tests)
Total Test time (real) = 1426.78 sec
The following tests did not run:
539 - Unit_hipMallocManaged_HostDeviceConcurrent (Skipped)
540 - Unit_hipMallocManaged_MultiChunkSingleDevice (Skipped)
541 - Unit_hipMallocManaged_MultiChunkMultiDevice (Skipped)
542 - Unit_hipMallocManaged_OverSubscription (Skipped)
543 - Unit_hipMallocManaged_TwoPointers - int (Skipped)
544 - Unit_hipMallocManaged_TwoPointers - float (Skipped)
545 - Unit_hipMallocManaged_TwoPointers - double (Skipped)
571 - Unit_hipMallocManaged_FlgParam (Skipped)
572 - Unit_hipMallocManaged_AccessMultiStream (Skipped)
604 - Unit_hipMallocManaged_Advanced (Skipped)
681 - Unit_hipMemsetSync (Skipped)
682 - Unit_hipMemsetDSync - int8_t (Skipped)
683 - Unit_hipMemsetDSync - int16_t (Skipped)
684 - Unit_hipMemsetDSync - uint32_t (Skipped)
685 - Unit_hipMemset2DSync (Skipped)
686 - Unit_hipMemset3DSync (Skipped)
754 - Unit_hipDeviceTotalMem_NonSelectedDevice (Skipped)
759 - Unit_hipGetDeviceCount_HideDevices (Skipped)
766 - Unit_hipSetGetDevice_Positive_Threaded_Basic (Skipped)
768 - Unit_hipDeviceGetP2PAttribute_Basic (Skipped)
769 - Unit_hipDeviceGetP2PAttribute_Negative (Skipped)
770 - Unit_hipDeviceCanAccessPeer_positive (Skipped)
771 - Unit_hipDeviceCanAccessPeer_negative (Skipped)
772 - Unit_hipDeviceEnableDisablePeerAccess_positive (Skipped)
773 - Unit_hipDeviceEnablePeerAccess_negative (Skipped)
774 - Unit_hipDeviceDisablePeerAccess_negative (Skipped)
The following tests FAILED:
817 - syncthreadsExitedThreads (Timeout)
system info:
Clang/LLVM & SPIRV-translator : both from `https://github.com/CHIP-SPV/ ` branch `chipStar-llvm-17`
OS: Ubuntu 22.04.2 LTS
GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04
CPU: Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz
GPU: Intel(R) Data Center GPU Max 1550
GPU Driver Version 23.35.27191.40
RC3 was built with:
cmake -G Ninja -DCHIP_USE_INTEL_USM=ON -DCMAKE_INSTALL_PREFIX=/nfs/site/home/mbabej/0/INSTALL/chip_17_11rc3_strict -DLLVM_CONFIG_BIN=/nfs/site/home/mbabej/0/INSTALL/LLVM_17/bin/llvm-config -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS_RELEASE="-O0 -ggdb3 -march=native" /nfs/site/home/mbabej/0/source/chipStar
changing the CXX_FLAGS to -O2 -march=native -DNDEBUG
does not seem to make any difference
OK, a regression between RC1 and RC3, we need to investigate and block 1.1.
Can you also post the igc version and a copy of the stack trace @franz
IGC (libigc1) version: 1.0.14508.23-704~22.04
intel-opencl-icd version: 23.35.27191.40-775~22.04
example stack trace:
gdb-oneapi ./samples/cuda_samples/cuda-FDTD3d
Thread 1 "cuda-FDTD3d" received signal SIGSEGV, Segmentation fault.
0x0000155554c513fe in __GI___libc_free (mem=0x100000001) at ./malloc/malloc.c:3368
3368 ./malloc/malloc.c: No such file or directory.
(gdb) bt
#0 0x0000155554c513fe in __GI___libc_free (mem=0x100000001) at ./malloc/malloc.c:3368
#1 0x000015553fc46944 in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#2 0x000015553fc476f2 in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#3 0x000015553fc4ae78 in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#4 0x000015553fc4c6c5 in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#5 0x000015553f7cfa9b in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#6 0x000015553f7cfc0d in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#7 0x000015553f7a0995 in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#8 0x000015553f7a0bdd in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#9 0x000015553f7a6066 in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#10 0x000015553f7a616d in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#11 0x000015553f723e2b in ?? () from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#12 0x00001555554a0aae in cl::detail::Wrapper<_cl_kernel*>::~Wrapper() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#13 0x00001555554a17b3 in CHIPKernelOpenCL::~CHIPKernelOpenCL() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#14 0x00001555554a17d7 in CHIPKernelOpenCL::~CHIPKernelOpenCL() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#15 0x000015555542ed9b in chipstar::Module::~Module() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#16 0x00001555554a1620 in CHIPModuleOpenCL::~CHIPModuleOpenCL() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#17 0x0000155555430c6a in chipstar::Device::~Device() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#18 0x00001555554a171d in CHIPDeviceOpenCL::~CHIPDeviceOpenCL() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#19 0x00001555554a165d in CHIPContextOpenCL::~CHIPContextOpenCL() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#20 0x00001555554a16a7 in CHIPContextOpenCL::~CHIPContextOpenCL() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#21 0x0000155555434e39 in chipstar::Backend::~Backend() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#22 0x00001555554a18b5 in CHIPBackendOpenCL::~CHIPBackendOpenCL() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#23 0x000015555542a980 in CHIPUninitializeCallOnce() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#24 0x0000155554c45ee8 in __pthread_once_slow (once_control=0x155555517518 <Uninitialized>, init_routine=0x155554eaed50 <__once_proxy>)
at ./nptl/pthread_once.c:116
#25 0x000015555542ae45 in void std::call_once<void (*)()>(std::once_flag&, void (*&&)()) ()
from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#26 0x000015555542a806 in CHIPUninitialize() () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#27 0x0000155555477d88 in __hipUnregisterFatBinary () from /nfs/site/home/mbabej/0/build/b_chip_17_FULL_BUILD/libCHIP.so
#28 0x0000555555556ae2 in __hip_module_dtor ()
#29 0x0000155554bf1495 in __run_exit_handlers (status=0, listp=0x155554dc5838 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true,
run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:113
#30 0x0000155554bf1610 in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:143
#31 0x0000555555557c8a in main ()
Is the crash similar for the super simple tests like the Unit_deviceFunctions_CompileTest___fmul_rn_float
?
And is the freed pointer always something that looks like a sentinel/dummy? mem=0x100000001
Is git bisect
helpful here?
finished git bisect: 228281b98eef35e2e7bd6ebbb1ae2d35d46590bb is the first bad commit "Add capability based HIP device library link"
I've checked with different driver versions; 23.30.26918.19
and 23.30.26918.20
do not crash,; 23.30.26918.28
, 23.30.26918.50
, and 23.35.27191.29
do crash.