[Bug]: Invalid Device Ordinal on ROCm

[Bellk17]

Your current environment

PyTorch version: 2.4.0.dev20240415+rocm6.0
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 6.0.32830-d62f6a171

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.2
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-102-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: AMD Instinct MI300X (gfx942:sramecc+:xnack-)
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 6.0.32830
MIOpen runtime version: 3.0.0
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      52 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             384
On-line CPU(s) list:                0-383
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 9654 96-Core Processor
CPU family:                         25
Model:                              17
Thread(s) per core:                 2
Core(s) per socket:                 96
Socket(s):                          2
Stepping:                           1
Frequency boost:                    enabled
CPU max MHz:                        3707.8120
CPU min MHz:                        1500.0000
BogoMIPS:                           4792.43
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                     AMD-V
L1d cache:                          6 MiB (192 instances)
L1i cache:                          6 MiB (192 instances)
L2 cache:                           192 MiB (192 instances)
L3 cache:                           768 MiB (24 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-95,192-287
NUMA node1 CPU(s):                  96-191,288-383
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pytorch-triton-rocm==3.0.0+0a22a91d04
[pip3] torch==2.4.0.dev20240415+rocm6.0
[pip3] torchvision==0.19.0.dev20240415+rocm6.0
[conda] No relevant packagesROCM Version: 6.1.33591-3a954afdc
Neuron SDK Version: N/A
vLLM Version: 0.4.0.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
Could not collect

🐛 Describe the bug

Issue with invalid ordinal when running with tp=2 on ROCm:

python benchmarks/benchmark_throughput.py --input-len=50 --output-len=100 --model=mistralai/Mistral-7B-v0.1 --tensor-parallel-size=2 --enforce-eager

Namespace(backend='vllm', dataset=None, input_len=50, output_len=100, model='mistralai/Mistral-7B-v0.1', tokenizer='mistralai/Mistral-7B-v0.1', quantization=None, tensor_parallel_size=2, n=1, use_beam_search=False, num_prompts=1000, seed=0, hf_max_batch_size=None, trust_remote_code=False, max_model_len=None, dtype='auto', gpu_memory_utilization=0.9, enforce_eager=True, kv_cache_dtype='auto', quantization_param_path=None, device='cuda', enable_prefix_caching=False, enable_chunked_prefill=False, max_num_batched_tokens=None, download_dir=None)
INFO 04-16 23:49:56 pynccl.py:58] Loading nccl from library librccl.so.1
INFO 04-16 23:49:56 config.py:523] Disabled the custom all-reduce kernel because it is not supported on AMD GPUs.
2024-04-16 23:49:58,704 INFO worker.py:1724 -- Started a local Ray instance.
INFO 04-16 23:50:00 llm_engine.py:87] Initializing an LLM engine (v0.4.0.post1) with config: model='mistralai/Mistral-7B-v0.1', speculative_config=None, tokenizer='mistralai/Mistral-7B-v0.1', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0)
(pid=1376632) INFO 04-16 23:50:01 pynccl.py:58] Loading nccl from library librccl.so.1
INFO 04-16 23:50:04 selector.py:38] Using ROCmFlashAttention backend.
(RayWorkerVllm pid=1376787) INFO 04-16 23:50:04 selector.py:38] Using ROCmFlashAttention backend.
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50] Error executing method init_device. This might cause deadlock in distributed execution.
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50] Traceback (most recent call last):
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50]   File "/home/vllm_install/vllm/vllm/engine/ray_utils.py", line 43, in execute_method
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50]     return executor(*args, **kwargs)
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50]   File "/home/vllm_install/vllm/vllm/worker/worker.py", line 97, in init_device
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50]     torch.cuda.set_device(self.device)
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50]   File "/home/vllm_install/venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 399, in set_device
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50]     torch._C._cuda_setDevice(device)
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50] RuntimeError: HIP error: invalid device ordinal
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50] HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50] For debugging consider passing AMD_SERIALIZE_KERNEL=3.
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50] Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
(RayWorkerVllm pid=1376787) ERROR 04-16 23:50:04 ray_utils.py:50] 

This with building on latest main. Was hoping this was fixed with https://github.com/vllm-project/vllm/pull/3770, but no amount of environmental configuration has helped either (CUDA_VISIBLE_DEVICES, etc).

[ehartford]

I get same error

(vllm) ehartford@tw003:~/models/dolphin-2.9.2-qwen2-72b$ python -m vllm.entrypoints.openai.api_server --trust-remote-code --tensor-parallel-size 8 --model /home/ehartford/models/dolphin-2.9.2-qwen2-72b
INFO 05-27 20:36:35 config.py:569] Disabled the custom all-reduce kernel because it is not supported on AMD GPUs.
2024-05-27 20:36:37,825 INFO worker.py:1749 -- Started a local Ray instance.
INFO 05-27 20:36:40 llm_engine.py:103] Initializing an LLM engine (v0.4.2) with config: model='/home/ehartford/models/dolphin-2.9.2-qwen2-72b', speculative_config=None, tokenizer='/home/ehartford/models/dolphin-2.9.2-qwen2-72b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/home/ehartford/models/dolphin-2.9.2-qwen2-72b)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 05-27 20:36:57 selector.py:56] Using ROCmFlashAttention backend.
(RayWorkerWrapper pid=2194398) INFO 05-27 20:36:57 selector.py:56] Using ROCmFlashAttention backend.
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146] Error executing method init_device. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146] Traceback (most recent call last):
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146]   File "/home/ehartford/miniconda3/envs/vllm/lib/python3.11/site-packages/vllm-0.4.2+rocm613-py3.11-linux-x86_64.egg/vllm/worker/worker_base.py", line 138, in execute_method
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146]     return executor(*args, **kwargs)
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146]   File "/home/ehartford/miniconda3/envs/vllm/lib/python3.11/site-packages/vllm-0.4.2+rocm613-py3.11-linux-x86_64.egg/vllm/worker/worker.py", line 105, in init_device
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146]     torch.cuda.set_device(self.device)
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146]   File "/home/ehartford/miniconda3/envs/vllm/lib/python3.11/site-packages/torch/cuda/__init__.py", line 404, in set_device
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146]     torch._C._cuda_setDevice(device)
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146] RuntimeError: HIP error: invalid device ordinal
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146] HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146] For debugging consider passing HIP_LAUNCH_BLOCKING=1.
(RayWorkerWrapper pid=2194149) ERROR 05-27 20:36:57 worker_base.py:146] Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

[linchen111]

did you solve this?

[fzp0424]

ehartford

Hello Eric, I just encountered the same issue as you. Have you resolved it?

[fzp0424]

did you solve this?

Did you solve this? Several of my machine encountered this issue at the same time.

[fzp0424]

did you solve this?

Did you solve this? Several of my machine encountered this issue at the same time.

Reinstalling the conda env can solve this.

[linchen111]

did you solve this?

Did you solve this? Several of my machine encountered this issue at the same time.

Reinstalling the conda env can solve this.

how to do this?

[ehartford]

did you solve this?

Did you solve this? Several of my machine encountered this issue at the same time.

Reinstalling the conda env can solve this.

I will try this if I encounter it again

[hongxiayang]

Do you still have the issue? Please update.

[hongxiayang]

Closing this issue as the issue should have been resolved. Please open a new one if you run into the similar issue again