P2P Access Not Working on Level Zero

kaanolgu commented 4 days ago

Describe the bug

Although documented here : https://github.com/intel/llvm/blob/ab6c0f5e7ac81f406f5e6ab6875e4c4e6a7ab906/sycl/test-e2e/USM/P2P/p2p_access.cpp#L1

It doesn't work with level_zero is there something else I need to enable or include ?

To reproduce

export SYCL_PI_TRACE=1 export SYCL_PRINT_EXECUTION_GRAPH="always" export ONEAPI_DEVICE_SELECTOR=level_zero:gpu

./build/run.exe

Caught a SYCL host exception:
Native API failed. Native API returns: -995 (The plugin or device does not support the called function) -995 (The plugin or device does not support the called function)
terminate called after throwing an instance of 'sycl::_V1::runtime_error'
  what():  Native API failed. Native API returns: -995 (The plugin or device does not support the called function) -995 (The plugin or device does not support the called function)

Environment

OS: [e.g Windows/Linux]
Target device and vendor: Intel(R) Data Center GPU Max 1550
DPC++ version: $ icpx --version Intel(R) oneAPI DPC++/C++ Compiler 2024.2.1 (2024.2.1.20240711) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/intel/oneapi/compiler/2024.2/bin/compiler Configuration file: /opt/intel/oneapi/compiler/2024.2/bin/compiler/../icpx.cfg

Dependencies version:


$ sycl-ls --verbose
INFO: Output filtered by ONEAPI_DEVICE_SELECTOR environment variable, which is set to level_zero:gpu.
To see device ids, use the --ignore-device-selectors CLI option.

[level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] [level_zero:gpu] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191]

Platforms: 1 Platform [#1]: Version : 1.3 Name : Intel(R) Level-Zero Vendor : Intel(R) Corporation Devices : 16 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 Type : gpu Version : 1.3 Name : Intel(R) Data Center GPU Max 1550 Vendor : Intel(R) Corporation Driver : 1.3.27191 Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_bindless_images ext_oneapi_bindless_images_shared_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_mipmap ext_oneapi_mipmap_anisotropy ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_graph info::device::sub_group_sizes: 16 32 default_selector() : gpu, Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] accelerator_selector() : No device of requested type available. Please chec... cpu_selector() : No device of requested type available. Please chec... gpu_selector() : gpu, Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] custom_selector(gpu) : gpu, Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.27191] custom_selector(cpu) : No device of requested type available. Please chec... custom_selector(acc) : No device of requested type available. Please chec...



### Additional context

_No response_

kaanolgu commented 4 days ago

https://github.com/intel/llvm/pull/14095

I am confused now, is p2p_access available for level_zero or not ?

JackAKirk commented 3 days ago

cc @nrspruit

JackAKirk commented 2 days ago

See https://github.com/intel/llvm/pull/15847#issuecomment-2437443907 I'll try to fix the implementation.

JackAKirk commented 2 days ago

@kaanolgu do you want to check if https://github.com/intel/llvm/pull/15873 fixes your issue using this branch https://github.com/JackAKirk/llvm/tree/p2p-fix-query? I don't have access to L0 machines but I suspect it might fix your issue.

kaanolgu commented 1 day ago

Hi @JackAKirk , Yes it solved the issue in other thread and I tried this nightly build p2p access and sycl-ls also worked https://github.com/intel/llvm/releases/tag/nightly-2024-10-25 so this clue hopefully help ? Thank you again

intel / llvm