search

intel / llvm

Intel staging area for llvm.org contribution. Home for Intel LLVM-based projects.
Other
1.25k stars 738 forks source link

Error building with NVidia CUDA #10655

Closed Ashutosh-Londhe closed 1 month ago

Ashutosh-Londhe commented 1 year ago

Describe the bug I was getting an error while building a sycl with NVidia CUDA

I followed the procedure mentioned on getting started guide

export DPCPP_HOME=~/install/sycl
mkdir $DPCPP_HOME
cd $DPCPP_HOME
git clone https://github.com/intel/llvm -b sycl

#loaded required module for cmake and python

CUDA_LIB_PATH=/opt/cuda/10.2.89/toolkit/lib64/stubs CC=gcc CXX=g++ python $DPCPP_HOME/llvm/buildbot/configure.py --cuda --cmake-opt="-DCUDA_TOOLKIT_ROOT_DIR=/opt/cuda/10.2.89/toolkit"
CUDA_LIB_PATH=/opt/cuda/10.2.89/toolkit/lib64/stubs CC=gcc CXX=g++ python $DPCPP_HOME/llvm/buildbot/compile.py

but when i ran compile.py file

i am getting following error

[4/2904] Building CXX object tools/sycl/plugins/unified_runtime/CMakeFiles/ur_adapter_cuda.dir/ur/adapters/cuda/image.cpp.o
FAILED: tools/sycl/plugins/unified_runtime/CMakeFiles/ur_adapter_cuda.dir/ur/adapters/cuda/image.cpp.o
/usr/bin/g++ -DENABLE_OPAQUE_POINTERS=1 -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GNU_SOURCE -D_LIBCPP_ENABLE_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__SYCL_BUILD_SYCL_DLL -Dur_adapter_cuda_EXPORTS -Itools/sycl/plugins/unified_runtime -I/ext-home/asl/install/sycl/llvm/sycl/plugins/unified_runtime -Iinclude -I/ext-home/asl/install/sycl/llvm/llvm/include -I/ext-home/asl/install/sycl/llvm/sycl/include -I_deps/unified-runtime-src/include -isystem /opt/cuda/10.2.89/toolkit/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -fno-lifetime-dse -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment -Wno-misleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -Wextra -Wno-deprecated-declarations -O3 -DNDEBUG -fPIC -UNDEBUG -fvisibility=hidden -std=c++17 -MD -MT tools/sycl/plugins/unified_runtime/CMakeFiles/ur_adapter_cuda.dir/ur/adapters/cuda/image.cpp.o -MF tools/sycl/plugins/unified_runtime/CMakeFiles/ur_adapter_cuda.dir/ur/adapters/cuda/image.cpp.o.d -o tools/sycl/plugins/unified_runtime/CMakeFiles/ur_adapter_cuda.dir/ur/adapters/cuda/image.cpp.o -c /ext-home/asl/install/sycl/llvm/sycl/plugins/unified_runtime/ur/adapters/cuda/image.cpp
/ext-home/asl/install/sycl/llvm/sycl/plugins/unified_runtime/ur/adapters/cuda/image.cpp: In function ‘ur_result_t urToCudaImageChannelFormat(ur_image_channel_type_t, CUarray_format*, size_t*)’:
/ext-home/asl/install/sycl/llvm/sycl/plugins/unified_runtime/ur/adapters/cuda/image.cpp:74:44: error: ‘CU_AD_FORMAT_UNORM_INT8X1’ was not declared in this scope
     CASE(UR_IMAGE_CHANNEL_TYPE_UNORM_INT8, CU_AD_FORMAT_UNORM_INT8X1, 1)
                                            ^~~~~~~~~~~~~~~~~~~~~~~~~
/ext-home/asl/install/sycl/llvm/sycl/plugins/unified_runtime/ur/adapters/cuda/image.cpp:70:19: note: in definition of macro ‘CASE’
     cuda_format = TO;                                                          \
                   ^~
/ext-home/asl/install/sycl/llvm/sycl/plugins/unified_runtime/ur/adapters/cuda/image.cpp:74:44: note: suggested alternative: ‘CU_AD_FORMAT_SIGNED_INT8’
     CASE(UR_IMAGE_CHANNEL_TYPE_UNORM_INT8, CU_AD_FORMAT_UNORM_INT8X1, 1)
                                            ^~~~~~~~~~~~~~~~~~~~~~~~~
/ext-home/asl/install/sycl/llvm/sycl/plugins/unified_runtime/ur/adapters/cuda/image.cpp:70:19: note: in definition of macro ‘CASE’
     cuda_format = TO;                                                          \
                   ^~
/ext-home/asl/install/sycl/llvm/sycl/plugins/unified_runtime/ur/adapters/cuda/image.cpp:77:45: error: ‘CU_AD_FORMAT_UNORM_INT16X1’ was not declared in this scope

In my case , the CUDA is not installed on standard path, so as mentioned on getting started i didnt added it to path and ld_library_path

JackAKirk commented 1 year ago

Recent changes to Images means that the CUDA backend requires the nvidia runtime toolkit with version >= 11.5 I am in the middle of updating the gettingStartedGuide.md to reflect this. If you use a toolkit version >=11.5 then this should resolve your issue.

Ashutosh-Londhe commented 1 year ago

Hi Jack,

I try bulding with CUDA toolkit 12.0 and able to build it But when i tried buliding and running the test application mentioned on Getting started guide, I am able to build, but getting following errors

`clang++ -fsycl simple-sycl-app.cpp -o simple-sycl-app.exe

 ./simple-sycl-app.exe
terminate called after throwing an instance of 'sycl::_V1::runtime_error'
  what():  No device of requested type available. -1 (PI_ERROR_DEVICE_NOT_FOUND)
Aborted
`
`clang++ -fsycl -fsycl-targets=nvptx64-nvidia-cuda   simple-sycl-app.cpp -o simple-sycl-app-cuda.exe --cuda-path=/opt/nvidia/hpc_sdk/Linux_x86_64/23.1/cuda/12.0

ONEAPI_DEVICE_SELECTOR=cuda:*  ./simple-sycl-app-cuda.exe
terminate called after throwing an instance of 'sycl::_V1::runtime_error'
  what():  No device of requested type available. -1 (PI_ERROR_DEVICE_NOT_FOUND)
Aborted
`

without cuda one should run on CPU right?

JackAKirk commented 1 year ago

Are you getting sensible results from

$ nvidia-smi.exe

? (if nvidia-smi.exe is not found in your path then you'll have to find where it is located - it is installed somewhere by the cuda toolkit)

Ashutosh-Londhe commented 1 year ago

Yes Jack

here is the output from Nvidia-smi

`nvidia-smi
Fri Aug  4 13:08:40 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  On   | 00000000:86:00.0 Off |                    0 |
| N/A   30C    P0    24W / 250W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+`
JackAKirk commented 1 year ago

Yes Jack

here is the output from Nvidia-smi

`nvidia-smi
Fri Aug  4 13:08:40 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  On   | 00000000:86:00.0 Off |                    0 |
| N/A   30C    P0    24W / 250W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+`

This looks problematic. It states that the cuda version of the installed driver and toolkit that is in your path is 11.2, but you have apparently used 12.0 to build intel/llvm. I think you need to make sure you are using 12.0 and a compatible driver. I think you may have to uninstall the existing driver that is apparently for toolkit 11.2. Then install an appropriate driver for 12.0 and your system: https://www.nvidia.com/Download/index.aspx?lang=en-us.

I have been able to build and run simple-sycl-app.cpp fine myself using the build/run instructions you have used.

Ashutosh-Londhe commented 1 year ago

Thanks Jack, i will do the needful

JackAKirk commented 1 year ago

Hi Jack, Do you mean update the CUDA driver on GPU to 12.0??

I've been thinking about this. It certainly works in many cases that you can use a new toolkit with an earlier driver version. It may be that in this case you do have to update the driver version, but to be honest I'm not certain that is your issue. Perhaps there is some other issue with your build.

If you tell me the complete steps you took to install dpc++ that would help?

JackAKirk commented 1 year ago

Thanks Jack, i will do the needful

can you do 

echo $LD_LIBRARY_PATH

and also do

SYCL_PI_TRACE=-1 ./simple-sycl-app.exe

and post the results?

Hopefully this will provide more information, Thanks.

Ashutosh-Londhe commented 1 year ago 

`echo $LD_LIBRARY_PATH
/ext-home/asl/install/sycl/llvm/build/lib:
asl@telos:~/install/sycl$ SYCL_PI_TRACE=-1 ./simple-sycl-app.exe
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_opencl.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_level_zero.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_cuda.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_cuda.so: undefined symbol: cuDeviceGetUui                                     d_v2>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_cuda.so
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_hip.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_hip.so: cannot open shared object file: No                                      such file or directory>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_hip.so
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_unified_runtime.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_native_cpu.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_native_cpu.so: cannot open shared o                                     bject file: No such file or directory>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_native_cpu.so
---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b7412c
) --->  pi_result : PI_SUCCESS

---> piPlatformsGet(
        <unknown> : 2
        <unknown> : 0xb02de0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb02de0[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 135168
        <unknown> : 4
        <unknown> : 0x7ffd02b7408c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb31310
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb31310
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b73f9c
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb02c50
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb02c50[ 0xaef578 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4096
        <unknown> : 8
        <unknown> : 0xb16448
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4162
        <unknown> : 8
        <unknown> : 0xb16450
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb16450[ 0 ... ]

---> piDeviceRetain(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73cb8
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 623
        <char * > : 0xb2fd90
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 135168
        <unknown> : 4
        <unknown> : 0x7ffd02b7408c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 2306
        <unknown> : 12
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: NVIDIA CUDA OpenCL platform found but is not compatible.
---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b7412c
) --->  pi_result : PI_SUCCESS

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b741ec
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb1fe90
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb1fe90[ 0xaef578 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4096
        <unknown> : 8
        <unknown> : 0xb16098
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4162
        <unknown> : 8
        <unknown> : 0xb160a0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb160a0[ 0 ... ]

---> piDeviceRetain(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73f08
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 623
        <char * > : 0xb30010
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Requested device_type: info::device_type::automatic
---> piextDeviceSelectBinary(
        <unknown> : 0xaef578
        <unknown> : 0x7ffd02b74138
        <unknown> : 1
        <unknown> : 0x7ffd02b74134
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74138[ 0x4129d0 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4145
        <unknown> : 8
        <unknown> : 0x7ffd02b74188
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74188[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b741a8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b74198
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 41
        <char * > : 0xb25b90
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Candidate device: -> score = 1300
SYCL_PI_TRACE[all]:   platform: Intel(R) OpenCL
SYCL_PI_TRACE[all]:   device: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4145
        <unknown> : 8
        <unknown> : 0x7ffd02b74188
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74188[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b741a8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b74198
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 41
        <char * > : 0xb14820
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Selected device: -> final score = 1300
SYCL_PI_TRACE[all]:   platform: Intel(R) OpenCL
SYCL_PI_TRACE[all]:   device: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b740ec
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb1feb0[ 0xaef578 ... ]

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piContextCreate(
        <nullptr>
        <unknown> : 1
        <unknown> : 0xb26780
        <nullptr>
        <nullptr>
        <unknown> : 0xb30378
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb26780[ 0xaef578 ... ]
        [out]<unknown> ** : 0xb30378[ 0xb30678 ... ]

---> piextQueueCreate(
        <unknown> : 0xb30678
        <unknown> : 0xaef578
        <unknown> : 0x7ffd02b740e0
        <unknown> : 0x7ffd02b740b8
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b740b8[ 0xb162f8 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4149
        <unknown> : 4
        <unknown> : 0x7ffd02b72c3c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piMemBufferCreate(
        <unknown> : 0xb30678
        <unknown> : 1
        <unknown> : 32
        <unknown> : 0
        <unknown> : 0x7ffd02b72d10
        <unknown> : 0
) --->  pi_result : PI_SUCCESS
        [out]void * : 0
        [out]pi_mem * : 0x7ffd02b72d10[ 0x2087878 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 65576
        <unknown> : 4
        <unknown> : 0x7ffd02b725bc
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piextDeviceSelectBinary(
        <unknown> : 0xaef578
        <unknown> : 0x2088050
        <unknown> : 1
        <unknown> : 0x7ffd02b724dc
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x2088050[ 0x412910 ... ]

---> piPluginGetBackendOption(
        pi_platform : 0xb00d58
        <const char *>: -O2
        <unknown> : 0x7ffd02b72350
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b72350[  ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4136
        <unknown> : 4
        <unknown> : 0x7ffd02b7236c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2305
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b722e8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2305
        <unknown> : 17
        <char * > : 0x2088070
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piProgramCreate(
        <unknown> : 0xb30678
        <unknown> : 0x412a60
        <unknown> : 5552
        <unknown> : 0x7ffd02b72368
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b72368[ 0x20881c8 ... ]

---> piProgramBuild(
        <unknown> : 0x20881c8
        <unknown> : 1
        <unknown> : 0xb16090
        <const char *>:
        <nullptr>
        <nullptr>
InvalidModule: Invalid SPIR-V module: input SPIR-V module uses unknown extension 'SPV_KHR_expect_assume'``echo $LD_LIBRARY_PATH
/ext-home/asl/install/sycl/llvm/build/lib:
asl@telos:~/install/sycl$ SYCL_PI_TRACE=-1 ./simple-sycl-app.exe
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_opencl.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_level_zero.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_cuda.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_cuda.so: undefined symbol: cuDeviceGetUui                                     d_v2>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_cuda.so
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_hip.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_hip.so: cannot open shared object file: No                                      such file or directory>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_hip.so
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_unified_runtime.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_native_cpu.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_native_cpu.so: cannot open shared o                                     bject file: No such file or directory>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_native_cpu.so
---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b7412c
) --->  pi_result : PI_SUCCESS

---> piPlatformsGet(
        <unknown> : 2
        <unknown> : 0xb02de0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb02de0[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 135168
        <unknown> : 4
        <unknown> : 0x7ffd02b7408c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb31310
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb31310
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b73f9c
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb02c50
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb02c50[ 0xaef578 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4096
        <unknown> : 8
        <unknown> : 0xb16448
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4162
        <unknown> : 8
        <unknown> : 0xb16450
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb16450[ 0 ... ]

---> piDeviceRetain(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73cb8
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 623
        <char * > : 0xb2fd90
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 135168
        <unknown> : 4
        <unknown> : 0x7ffd02b7408c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 2306
        <unknown> : 12
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: NVIDIA CUDA OpenCL platform found but is not compatible.
---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b7412c
) --->  pi_result : PI_SUCCESS

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b741ec
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb1fe90
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb1fe90[ 0xaef578 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4096
        <unknown> : 8
        <unknown> : 0xb16098
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4162
        <unknown> : 8
        <unknown> : 0xb160a0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb160a0[ 0 ... ]

---> piDeviceRetain(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73f08
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 623
        <char * > : 0xb30010
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Requested device_type: info::device_type::automatic
---> piextDeviceSelectBinary(
        <unknown> : 0xaef578
        <unknown> : 0x7ffd02b74138
        <unknown> : 1
        <unknown> : 0x7ffd02b74134
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74138[ 0x4129d0 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4145
        <unknown> : 8
        <unknown> : 0x7ffd02b74188
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74188[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b741a8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b74198
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 41
        <char * > : 0xb25b90
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Candidate device: -> score = 1300
SYCL_PI_TRACE[all]:   platform: Intel(R) OpenCL
SYCL_PI_TRACE[all]:   device: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4145
        <unknown> : 8
        <unknown> : 0x7ffd02b74188
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74188[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b741a8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b74198
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 41
        <char * > : 0xb14820
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Selected device: -> final score = 1300
SYCL_PI_TRACE[all]:   platform: Intel(R) OpenCL
SYCL_PI_TRACE[all]:   device: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b740ec
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb1feb0[ 0xaef578 ... ]

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piContextCreate(
        <nullptr>
        <unknown> : 1
        <unknown> : 0xb26780
        <nullptr>
        <nullptr>
        <unknown> : 0xb30378
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb26780[ 0xaef578 ... ]
        [out]<unknown> ** : 0xb30378[ 0xb30678 ... ]

---> piextQueueCreate(
        <unknown> : 0xb30678
        <unknown> : 0xaef578
        <unknown> : 0x7ffd02b740e0
        <unknown> : 0x7ffd02b740b8
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b740b8[ 0xb162f8 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4149
        <unknown> : 4
        <unknown> : 0x7ffd02b72c3c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piMemBufferCreate(
        <unknown> : 0xb30678
        <unknown> : 1
        <unknown> : 32
        <unknown> : 0
        <unknown> : 0x7ffd02b72d10
        <unknown> : 0
) --->  pi_result : PI_SUCCESS
        [out]void * : 0
        [out]pi_mem * : 0x7ffd02b72d10[ 0x2087878 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 65576
        <unknown> : 4
        <unknown> : 0x7ffd02b725bc
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piextDeviceSelectBinary(
        <unknown> : 0xaef578
        <unknown> : 0x2088050
        <unknown> : 1
        <unknown> : 0x7ffd02b724dc
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x2088050[ 0x412910 ... ]

---> piPluginGetBackendOption(
        pi_platform : 0xb00d58
        <const char *>: -O2
        <unknown> : 0x7ffd02b72350
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b72350[  ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4136
        <unknown> : 4
        <unknown> : 0x7ffd02b7236c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2305
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b722e8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2305
        <unknown> : 17
        <char * > : 0x2088070
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piProgramCreate(
        <unknown> : 0xb30678
        <unknown> : 0x412a60
        <unknown> : 5552
        <unknown> : 0x7ffd02b72368
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b72368[ 0x20881c8 ... ]

---> piProgramBuild(
        <unknown> : 0x20881c8
        <unknown> : 1
        <unknown> : 0xb16090
        <const char *>:
        <nullptr>
        <nullptr>
InvalidModule: Invalid SPIR-V module: input SPIR-V module uses unknown extension 'SPV_KHR_expect_assume'`
````echo $LD_LIBRARY_PATH
/ext-home/asl/install/sycl/llvm/build/lib:
asl@telos:~/install/sycl$ SYCL_PI_TRACE=-1 ./simple-sycl-app.exe
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_opencl.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_level_zero.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_cuda.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_cuda.so: undefined symbol: cuDeviceGetUui                                     d_v2>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_cuda.so
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_hip.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_hip.so: cannot open shared object file: No                                      such file or directory>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_hip.so
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_unified_runtime.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_native_cpu.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_native_cpu.so: cannot open shared o                                     bject file: No such file or directory>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_native_cpu.so
---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b7412c
) --->  pi_result : PI_SUCCESS

---> piPlatformsGet(
        <unknown> : 2
        <unknown> : 0xb02de0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb02de0[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 135168
        <unknown> : 4
        <unknown> : 0x7ffd02b7408c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb31310
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb31310
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b73f9c
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb02c50
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb02c50[ 0xaef578 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4096
        <unknown> : 8
        <unknown> : 0xb16448
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4162
        <unknown> : 8
        <unknown> : 0xb16450
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb16450[ 0 ... ]

---> piDeviceRetain(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73cb8
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 623
        <char * > : 0xb2fd90
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 135168
        <unknown> : 4
        <unknown> : 0x7ffd02b7408c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 2306
        <unknown> : 12
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: NVIDIA CUDA OpenCL platform found but is not compatible.
---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b7412c
) --->  pi_result : PI_SUCCESS

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b741ec
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb1fe90
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb1fe90[ 0xaef578 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4096
        <unknown> : 8
        <unknown> : 0xb16098
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4162
        <unknown> : 8
        <unknown> : 0xb160a0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb160a0[ 0 ... ]

---> piDeviceRetain(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73f08
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 623
        <char * > : 0xb30010
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Requested device_type: info::device_type::automatic
---> piextDeviceSelectBinary(
        <unknown> : 0xaef578
        <unknown> : 0x7ffd02b74138
        <unknown> : 1
        <unknown> : 0x7ffd02b74134
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74138[ 0x4129d0 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4145
        <unknown> : 8
        <unknown> : 0x7ffd02b74188
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74188[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b741a8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b74198
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 41
        <char * > : 0xb25b90
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Candidate device: -> score = 1300
SYCL_PI_TRACE[all]:   platform: Intel(R) OpenCL
SYCL_PI_TRACE[all]:   device: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4145
        <unknown> : 8
        <unknown> : 0x7ffd02b74188
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74188[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b741a8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b74198
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 41
        <char * > : 0xb14820
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Selected device: -> final score = 1300
SYCL_PI_TRACE[all]:   platform: Intel(R) OpenCL
SYCL_PI_TRACE[all]:   device: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b740ec
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb1feb0[ 0xaef578 ... ]

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piContextCreate(
        <nullptr>
        <unknown> : 1
        <unknown> : 0xb26780
        <nullptr>
        <nullptr>
        <unknown> : 0xb30378
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb26780[ 0xaef578 ... ]
        [out]<unknown> ** : 0xb30378[ 0xb30678 ... ]

---> piextQueueCreate(
        <unknown> : 0xb30678
        <unknown> : 0xaef578
        <unknown> : 0x7ffd02b740e0
        <unknown> : 0x7ffd02b740b8
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b740b8[ 0xb162f8 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4149
        <unknown> : 4
        <unknown> : 0x7ffd02b72c3c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piMemBufferCreate(
        <unknown> : 0xb30678
        <unknown> : 1
        <unknown> : 32
        <unknown> : 0
        <unknown> : 0x7ffd02b72d10
        <unknown> : 0
) --->  pi_result : PI_SUCCESS
        [out]void * : 0
        [out]pi_mem * : 0x7ffd02b72d10[ 0x2087878 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 65576
        <unknown> : 4
        <unknown> : 0x7ffd02b725bc
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piextDeviceSelectBinary(
        <unknown> : 0xaef578
        <unknown> : 0x2088050
        <unknown> : 1
        <unknown> : 0x7ffd02b724dc
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x2088050[ 0x412910 ... ]

---> piPluginGetBackendOption(
        pi_platform : 0xb00d58
        <const char *>: -O2
        <unknown> : 0x7ffd02b72350
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b72350[  ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4136
        <unknown> : 4
        <unknown> : 0x7ffd02b7236c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2305
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b722e8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2305
        <unknown> : 17
        <char * > : 0x2088070
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piProgramCreate(
        <unknown> : 0xb30678
        <unknown> : 0x412a60
        <unknown> : 5552
        <unknown> : 0x7ffd02b72368
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b72368[ 0x20881c8 ... ]

---> piProgramBuild(
        <unknown> : 0x20881c8
        <unknown> : 1
        <unknown> : 0xb16090
        <const char *>:
        <nullptr>
        <nullptr>
InvalidModule: Invalid SPIR-V module: input SPIR-V module uses unknown extension 'SPV_KHR_expect_assume'``echo $LD_LIBRARY_PATH
/ext-home/asl/install/sycl/llvm/build/lib:
asl@telos:~/install/sycl$ SYCL_PI_TRACE=-1 ./simple-sycl-app.exe
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_opencl.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_level_zero.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_cuda.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_cuda.so: undefined symbol: cuDeviceGetUui                                     d_v2>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_cuda.so
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_hip.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_hip.so: cannot open shared object file: No                                      such file or directory>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_hip.so
SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_unified_runtime.so [ PluginVersion: 14.37.1 ]
SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_native_cpu.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_native_cpu.so: cannot open shared o                                     bject file: No such file or directory>
SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_native_cpu.so
---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b7412c
) --->  pi_result : PI_SUCCESS

---> piPlatformsGet(
        <unknown> : 2
        <unknown> : 0xb02de0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb02de0[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 135168
        <unknown> : 4
        <unknown> : 0x7ffd02b7408c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb31310
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb31310
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b73f9c
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb02c50
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb02c50[ 0xaef578 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4096
        <unknown> : 8
        <unknown> : 0xb16448
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4162
        <unknown> : 8
        <unknown> : 0xb16450
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb16450[ 0 ... ]

---> piDeviceRetain(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73cb8
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 623
        <char * > : 0xb2fd90
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 135168
        <unknown> : 4
        <unknown> : 0x7ffd02b7408c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73fe8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb1fd30
        <unknown> : 2306
        <unknown> : 12
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: NVIDIA CUDA OpenCL platform found but is not compatible.
---> piPlatformsGet(
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b7412c
) --->  pi_result : PI_SUCCESS

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b741ec
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb1fe90
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb1fe90[ 0xaef578 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4096
        <unknown> : 8
        <unknown> : 0xb16098
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4162
        <unknown> : 8
        <unknown> : 0xb160a0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb160a0[ 0 ... ]

---> piDeviceRetain(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b73f08
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4144
        <unknown> : 623
        <char * > : 0xb30010
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Requested device_type: info::device_type::automatic
---> piextDeviceSelectBinary(
        <unknown> : 0xaef578
        <unknown> : 0x7ffd02b74138
        <unknown> : 1
        <unknown> : 0x7ffd02b74134
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74138[ 0x4129d0 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4145
        <unknown> : 8
        <unknown> : 0x7ffd02b74188
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74188[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b741a8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b74198
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 41
        <char * > : 0xb25b90
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Candidate device: -> score = 1300
SYCL_PI_TRACE[all]:   platform: Intel(R) OpenCL
SYCL_PI_TRACE[all]:   device: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4145
        <unknown> : 8
        <unknown> : 0x7ffd02b74188
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b74188[ 0xb00d58 ... ]

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b741a8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2306
        <unknown> : 16
        <char * > : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b74198
) --->  pi_result : PI_SUCCESS

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4139
        <unknown> : 41
        <char * > : 0xb14820
        <nullptr>
) --->  pi_result : PI_SUCCESS

SYCL_PI_TRACE[all]: Selected device: -> final score = 1300
SYCL_PI_TRACE[all]:   platform: Intel(R) OpenCL
SYCL_PI_TRACE[all]:   device: Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 0
        <unknown> : 0
        <unknown> : 0x7ffd02b740ec
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0[ nullptr ]

---> piDevicesGet(
        pi_platform : 0xb00d58
        <unknown> : 4294967295
        <unknown> : 1
        <unknown> : 0xb1feb0
        <nullptr>
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb1feb0[ 0xaef578 ... ]

---> piDeviceRelease(
        <unknown> : 0xaef578
) --->  pi_result : PI_SUCCESS

---> piContextCreate(
        <nullptr>
        <unknown> : 1
        <unknown> : 0xb26780
        <nullptr>
        <nullptr>
        <unknown> : 0xb30378
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0xb26780[ 0xaef578 ... ]
        [out]<unknown> ** : 0xb30378[ 0xb30678 ... ]

---> piextQueueCreate(
        <unknown> : 0xb30678
        <unknown> : 0xaef578
        <unknown> : 0x7ffd02b740e0
        <unknown> : 0x7ffd02b740b8
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b740b8[ 0xb162f8 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4149
        <unknown> : 4
        <unknown> : 0x7ffd02b72c3c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piMemBufferCreate(
        <unknown> : 0xb30678
        <unknown> : 1
        <unknown> : 32
        <unknown> : 0
        <unknown> : 0x7ffd02b72d10
        <unknown> : 0
) --->  pi_result : PI_SUCCESS
        [out]void * : 0
        [out]pi_mem * : 0x7ffd02b72d10[ 0x2087878 ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 65576
        <unknown> : 4
        <unknown> : 0x7ffd02b725bc
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piextDeviceSelectBinary(
        <unknown> : 0xaef578
        <unknown> : 0x2088050
        <unknown> : 1
        <unknown> : 0x7ffd02b724dc
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x2088050[ 0x412910 ... ]

---> piPluginGetBackendOption(
        pi_platform : 0xb00d58
        <const char *>: -O2
        <unknown> : 0x7ffd02b72350
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b72350[  ... ]

---> piDeviceGetInfo(
        <unknown> : 0xaef578
        <unknown> : 4136
        <unknown> : 4
        <unknown> : 0x7ffd02b7236c
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2305
        <unknown> : 0
        <nullptr>
        <unknown> : 0x7ffd02b722e8
) --->  pi_result : PI_SUCCESS

---> piPlatformGetInfo(
        pi_platform : 0xb00d58
        <unknown> : 2305
        <unknown> : 17
        <char * > : 0x2088070
        <nullptr>
) --->  pi_result : PI_SUCCESS

---> piProgramCreate(
        <unknown> : 0xb30678
        <unknown> : 0x412a60
        <unknown> : 5552
        <unknown> : 0x7ffd02b72368
) --->  pi_result : PI_SUCCESS
        [out]<unknown> ** : 0x7ffd02b72368[ 0x20881c8 ... ]

---> piProgramBuild(
        <unknown> : 0x20881c8
        <unknown> : 1
        <unknown> : 0xb16090
        <const char *>:
        <nullptr>
        <nullptr>
InvalidModule: Invalid SPIR-V module: input SPIR-V module uses unknown extension 'SPV_KHR_expect_assume'`

This is what i get when dont add cuda libs to path
Ashutosh-Londhe commented 1 year ago

If i add CUDA libs to path then

echo $LD_LIBRARY_PATH /opt/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/nvshmem/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/nccl/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/23.1/comm_libs/mpi/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/23.1/math_libs/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/23.1/compilers/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/23.1/compilers/extras/qd/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/23.1/cuda/extras/CUPTI/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/23.1/cuda/lib64:/ext-home/asl/install/sycl/llvm/build/lib: asl@telos:~/install/sycl$ SYCL_PI_TRACE=-1 ./simple-sycl-app.exe SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_opencl.so) failed with </opt/nvidia/hpc_sdk/Linux_x86_64/23.1/cuda/lib64/libOpenCL.so.1: versionOPENCL_2.1' not found (required by /ext-home/asl/install/sycl/llvm/build/lib/libpi_opencl.so)> SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_opencl.so SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_level_zero.so [ PluginVersion: 14.37.1 ] SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_cuda.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_cuda.so: undefined symbol: cuDeviceGetUuid_v2> SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_cuda.so SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_hip.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_hip.so: cannot open shared object file: No such file or directory> SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_hip.so SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_unified_runtime.so [ PluginVersion: 14.37.1 ] SYCL_PI_TRACE[-1]: dlopen(/ext-home/asl/install/sycl/llvm/build/lib/libpi_native_cpu.so) failed with </ext-home/asl/install/sycl/llvm/build/lib/libpi_native_cpu.so: cannot open shared object file: No such file or directory> SYCL_PI_TRACE[all]: Check if plugin is present. Failed to load plugin: libpi_native_cpu.so ---> piPlatformsGet(

: 0 : 0x7ffd7e1483bc ) ---> pi_result : PI_SUCCESS terminate called after throwing an instance of 'sycl::_V1::runtime_error' what(): No device of requested type available. -1 (PI_ERROR_DEVICE_NOT_FOUND) Aborted`
JackAKirk commented 1 year ago

I see. You are using the intel/llvm sycl branch? what is the commit from your checked out intel/llvm sycl branch?

When you updated the cuda toolkit did you build intel/llvm from scratch by deleting the existing intel/llvm/build folder?

Ashutosh-Londhe commented 1 year ago

Yes Jack, i am using using sycl branch

latest commit is

`commit ec6ea1ca8b784601d21e2c849733389412bab999 (HEAD -> sycl, origin/sycl, origin/HEAD) Author: Vyacheslav Klochkov vyacheslav.n.klochkov@intel.com Date: Mon Jul 31 16:17:25 2023 -0500

[ESIMD][E2E] Fix flaky-ness of slm_gather_scatter.cpp test (#10629)

Signed-off-by: Vyacheslav N Klochkov <vyacheslav.n.klochkov@intel.com>`

and yes i did deleted llvm/build folder before starting configure
JackAKirk commented 1 year ago

Yes Jack, i am using using sycl branch

latest commit is

`commit ec6ea1c (HEAD -> sycl, origin/sycl, origin/HEAD) Author: Vyacheslav Klochkov vyacheslav.n.klochkov@intel.com Date: Mon Jul 31 16:17:25 2023 -0500

[ESIMD][E2E] Fix flaky-ness of slm_gather_scatter.cpp test (#10629)

Signed-off-by: Vyacheslav N Klochkov <vyacheslav.n.klochkov@intel.com>`

and yes i did deleted llvm/build folder before starting configure

Ah, try moving all libOpenCL*.so libs from your lib64 folder in your cuda 12.0 toolkit install. And make sure that this lib64 folder is in your LD_LIBRARY_PATH (this is a standard step in cuda toolkit installation).

This may be an incompatibility with your opencl dpc++ installation and cuda opencl libs.

Ashutosh-Londhe commented 1 year ago

I am actually working on cluster with non-sudo access so unfortunately cannot do this moving of OpenCL*.so out of lib63 folder

JackAKirk commented 1 year ago

I am actually working on cluster with non-sudo access so unfortunately cannot do this moving of OpenCL*.so out of lib63 folder

You can try a special ordering: add the path containing the correct OpenCL.so libs to LD_LIBRARY_PATH after adding the cuda lib64 path to LD_LIBRARY_PATH (or at least math sure that the correct OpenCL.so libs are found first.).

hdelan commented 1 year ago

It looks like you built DPC++ with a CUDA version newer than your driver version. See here https://github.com/intel/llvm/blob/sycl/sycl/plugins/unified_runtime/ur/adapters/cuda/device.cpp#L983C29-L983C29

Since CUDA toolkits don't have compatibility across major versions with the driver, you will need to build with a CUDA 11 toolkit if you want to keep your driver as CUDA 11. The issue that you have with the missing CU_AD_* is a bug which is hopefully fixed here https://github.com/intel/llvm/pull/10691

Can you checkout the branch in the PR and see if you can successfully build DPC++ with CUDA 11.x?

hdelan commented 1 year ago

10691 has been merged so this should be fixed now. Please let me know if this works now

abagusetty commented 1 year ago

@hdelan There are still a couple left that didn`t get fixed with the above PR: https://github.com/hdelan/llvm/blob/sycl/sycl/plugins/unified_runtime/ur/adapters/cuda/image.cpp#L127-L130

hdelan commented 1 year ago

Thanks @abagusetty here's the new patch https://github.com/intel/llvm/pull/10922

JackAKirk commented 1 month ago

Closed by https://github.com/intel/llvm/pull/10922