search

ROCm / clr

MIT License
104 stars 50 forks source link

[Issue]: CL_DEVICE_TOPOLOGY_TYPE_PCIE_AMD wrong on Adastra/MI300 because it cannot report PCI domains #106

Closed bgoglin closed 1 week ago

bgoglin commented 2 weeks ago

Problem Description

On MI300A nodes on the Adastra supercomputer, all 4 GPUs return the PCI BDF of the first GPU from the OpenCL PCIe query CL_DEVICE_TOPOLOGY_TYPE_PCIE_AMD

Operating System

RHEL 9.4

CPU

MI300A

GPU

MI300A

ROCm Version

ROCm 6.1.0

ROCm Component

clr

Steps to Reproduce

This was first discovered inside hwloc but can be confirmed from clinfo:

a1003:~/hwloc-3.0.0a1-git/build-mi300$ clinfo  | grep Topology
  Device Topology:               PCI[ B#2, D#0, F#0 ]
  Device Topology:               PCI[ B#2, D#0, F#0 ]
  Device Topology:               PCI[ B#2, D#0, F#0 ]
  Device Topology:               PCI[ B#2, D#0, F#0 ]

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

ROCm seems to be 6.1.2. Most rocm RHEL packages contain 60102-119 in their package name, including the opencl ones. rocm-opencl.x86_64 2.0.0.60102-119.el9 @other_rocm-6.1.2-rhel9_13
rocm-opencl-devel.x86_64 2.0.0.60102-119.el9 @other_rocm-6.1.2-rhel9_13
rocm-opencl-icd-loader.x86_64 1.2.60102-119.el9 @other_rocm-6.1.2-rhel9_13
rocm-opencl-runtime.x86_64 6.1.2.60102-119.el9 @other_rocm-6.1.2-rhel9_13
rocm-opencl-sdk.x86_64 6.1.2.60102-119.el9 @other_rocm-6.1.2-rhel9_13

ROCk module version 6.7.0 is loaded =====================
HSA System Attributes
=====================
Runtime Version: 1.13 Runtime Ext Version: 1.4 System Timestamp Freq.: 1000.000000MHz Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED DMAbuf Support: YES

==========
HSA Agents
==========

Agent 1

Name: AMD Instinct MI300A Accelerator
Uuid: CPU-XX
Marketing Name: AMD Instinct MI300A Accelerator
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3700
BDFID: 0
Internal Node ID: 0
Compute Unit: 48
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 130890508(0x7cd3b0c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED Size: 130890508(0x7cd3b0c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 130890508(0x7cd3b0c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

Agent 2

Name: AMD Instinct MI300A Accelerator
Uuid: CPU-XX
Marketing Name: AMD Instinct MI300A Accelerator
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 1
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3700
BDFID: 0
Internal Node ID: 1
Compute Unit: 48
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 131766468(0x7da98c4) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED Size: 131766468(0x7da98c4) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 131766468(0x7da98c4) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

Agent 3

Name: AMD Instinct MI300A Accelerator
Uuid: CPU-XX
Marketing Name: AMD Instinct MI300A Accelerator
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 2
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3700
BDFID: 0
Internal Node ID: 2
Compute Unit: 48
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 131809208(0x7db3fb8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED Size: 131809208(0x7db3fb8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 131809208(0x7db3fb8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

Agent 4

Name: AMD Instinct MI300A Accelerator
Uuid: CPU-XX
Marketing Name: AMD Instinct MI300A Accelerator
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 3
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3700
BDFID: 0
Internal Node ID: 3
Compute Unit: 48
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 131795632(0x7db0ab0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED Size: 131795632(0x7db0ab0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 131795632(0x7db0ab0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

Agent 5

Name: gfx942
Uuid: GPU-eb28776e0dc5fc14
Marketing Name: AMD Instinct MI300A
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 4
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 24576(0x6000) KB
L3: 262144(0x40000) KB
Chip ID: 29856(0x74a0)
ASIC Revision: 1(0x1)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2100
BDFID: 512
Internal Node ID: 4
Compute Unit: 228
SIMDs per CU: 4
Shader Engines: 24
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: TRUE
Features: KERNEL_DISPATCH Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension: x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 2048(0x800)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension: x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 138
SDMA engine uCode:: 19
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 4
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx942:sramecc+:xnack- Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension: x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension: x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32

Agent 6

Name: gfx942
Uuid: GPU-5f2efdd4cf3cf910
Marketing Name: AMD Instinct MI300A
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 5
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 24576(0x6000) KB
L3: 262144(0x40000) KB
Chip ID: 29856(0x74a0)
ASIC Revision: 1(0x1)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2100
BDFID: 512
Internal Node ID: 5
Compute Unit: 228
SIMDs per CU: 4
Shader Engines: 24
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: TRUE
Features: KERNEL_DISPATCH Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension: x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 2048(0x800)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension: x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 138
SDMA engine uCode:: 19
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 4
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx942:sramecc+:xnack- Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension: x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension: x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32

Agent 7

Name: gfx942
Uuid: GPU-65980e71d0ff713b
Marketing Name: AMD Instinct MI300A
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 6
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 24576(0x6000) KB
L3: 262144(0x40000) KB
Chip ID: 29856(0x74a0)
ASIC Revision: 1(0x1)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2100
BDFID: 512
Internal Node ID: 6
Compute Unit: 228
SIMDs per CU: 4
Shader Engines: 24
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: TRUE
Features: KERNEL_DISPATCH Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension: x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 2048(0x800)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension: x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 138
SDMA engine uCode:: 19
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 4
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx942:sramecc+:xnack- Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension: x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension: x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32

Agent 8

Name: gfx942
Uuid: GPU-ada4590817227e95
Marketing Name: AMD Instinct MI300A
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 7
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 24576(0x6000) KB
L3: 262144(0x40000) KB
Chip ID: 29856(0x74a0)
ASIC Revision: 1(0x1)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2100
BDFID: 512
Internal Node ID: 7
Compute Unit: 228
SIMDs per CU: 4
Shader Engines: 24
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: TRUE
Features: KERNEL_DISPATCH Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension: x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 2048(0x800)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension: x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 138
SDMA engine uCode:: 19
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 134217728(0x8000000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 4
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx942:sramecc+:xnack- Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension: x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension: x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
Done

Additional Information

ROCM-SMI correctly reports the PCI BDF of each GPU. The issue seems to be inside the OpenCL runtime.

Adastra/MI250X doesn't have the problem:

g1003:~$ clinfo | grep Topology
  Device Topology:               PCI[ B#193, D#0, F#0 ]
  Device Topology:               PCI[ B#198, D#0, F#0 ]
  Device Topology:               PCI[ B#201, D#0, F#0 ]
  Device Topology:               PCI[ B#206, D#0, F#0 ]
  Device Topology:               PCI[ B#209, D#0, F#0 ]
  Device Topology:               PCI[ B#214, D#0, F#0 ]
  Device Topology:               PCI[ B#217, D#0, F#0 ]
  Device Topology:               PCI[ B#222, D#0, F#0 ]

However the OS is a RHEL 8.8 and ROCm 5.7.1.

ppanchad-amd commented 2 weeks ago

Hi @bgoglin. Internal ticket is created to investigate your issue. Thanks!

schung-amd commented 1 week ago

Hi @bgoglin, what does rocm-smi --showcomputepartition show? It's possible that these four nodes are partitions of the same GPU, in which case they would have the same BDF. If this is not the case, to clarify, do you see the same BDF reported by rocm-smi --showbus and amd-smi topology?

bgoglin commented 1 week ago

@schung-amd Here's the output of all these commands:

$ rocm-smi --showcomputepartition

============================ ROCm System Management Interface ============================
=============================== Current Compute Partition ================================
GPU[0]      : Compute Partition: SPX
GPU[1]      : Compute Partition: SPX
GPU[2]      : Compute Partition: SPX
GPU[3]      : Compute Partition: SPX
==========================================================================================
================================== End of ROCm SMI Log ===================================
a1003:~$ rocm-smi --showbus

============================ ROCm System Management Interface ============================
======================================= PCI Bus ID =======================================
GPU[0]      : PCI Bus: 0000:02:00.0
GPU[1]      : PCI Bus: 0001:02:00.0
GPU[2]      : PCI Bus: 0002:02:00.0
GPU[3]      : PCI Bus: 0003:02:00.0
==========================================================================================
================================== End of ROCm SMI Log ===================================
a1003:~$ amd-smi topology
ACCESS TABLE:
             0000:02:00.0 0001:02:00.0 0002:02:00.0 0003:02:00.0
0000:02:00.0 ENABLED      ENABLED      ENABLED      ENABLED
0001:02:00.0 ENABLED      ENABLED      ENABLED      ENABLED
0002:02:00.0 ENABLED      ENABLED      ENABLED      ENABLED
0003:02:00.0 ENABLED      ENABLED      ENABLED      ENABLED

WEIGHT TABLE:
             0000:02:00.0 0001:02:00.0 0002:02:00.0 0003:02:00.0
0000:02:00.0 0            15           15           15
0001:02:00.0 15           0            15           15
0002:02:00.0 15           15           0            15
0003:02:00.0 15           15           15           0

HOPS TABLE:
             0000:02:00.0 0001:02:00.0 0002:02:00.0 0003:02:00.0
0000:02:00.0 0            1            1            1
0001:02:00.0 1            0            1            1
0002:02:00.0 1            1            0            1
0003:02:00.0 1            1            1            0

LINK TYPE TABLE:
             0000:02:00.0 0001:02:00.0 0002:02:00.0 0003:02:00.0
0000:02:00.0 SELF         XGMI         XGMI         XGMI
0001:02:00.0 XGMI         SELF         XGMI         XGMI
0002:02:00.0 XGMI         XGMI         SELF         XGMI
0003:02:00.0 XGMI         XGMI         XGMI         SELF

NUMA BW TABLE:
             0000:02:00.0 0001:02:00.0 0002:02:00.0 0003:02:00.0
0000:02:00.0 N/A          50000-100000 50000-100000 50000-100000
0001:02:00.0 50000-100000 N/A          50000-100000 50000-100000
0002:02:00.0 50000-100000 50000-100000 N/A          50000-100000
0003:02:00.0 50000-100000 50000-100000 50000-100000 N/A
schung-amd commented 1 week ago

Thanks! It looks like the GPUs are connected to different PCI domains which each have their own set of BDFs, so you have 0000:02:00.0, 0001:02:00.0, etc.

For some clarification,

ROCM-SMI correctly reports the PCI BDF of each GPU. The issue seems to be inside the OpenCL runtime.

Are you referring to rocm-smi --showbus output here, or did you see other values for the BDF from rocm-smi?

Is this causing any issues for you, or just a curiosity?

bgoglin commented 1 week ago

I was querying the PCI BDF from the ROCm SMI API rsmi_dev_pci_id_get(). Those values are correct. Only the one from the CL extension API was wrong. Both are queried by the hwloc library, which is used by some parallel runtime in upper layers to place tasks and memory near the GPU they use. It's not a huge issue right now, but it'd be nice to solve this.

So the issue is actually that the CL extension HWLOC_CL_DEVICE_TOPOLOGY_AMD doesn't report any PCI domain at all. One easy way to solve this would be to implement HWLOC_CL_DEVICE_PCI_BUS_INFO_KHR instead. This issue seems to be more standard, and it reports the PCI domain. Implementing this should be fairly easy on your side.

schung-amd commented 1 week ago

I agree, just verifying that it's only that CL API call reporting limited information and not our whole stack being wrong. As you say this should be relatively straightforward to address and would make sense in my opinion. I'll discuss this with the internal team, thanks for bringing this to our attention!

schung-amd commented 1 week ago

Unfortunately, we are no longer adding new features or functionality to our OpenCL implementation at this time, which this would fall under (as CL_DEVICE_PCI_BUS_INFO_KHR is a CL3.0 spec for example), so this is not likely to be addressed in the foreseeable future. I'm also not aware of any workarounds available on the OpenCL side; rocm-smi and amd-smi get their information from a completely different source so nothing can really be ported over. Regardless, this is good knowledge for similar issues that may arise in the future, so thank you for reporting this. Feel free to comment if you'd like additional guidance.