Open erichsu0527 opened 1 month ago
@vickytsang can you take a look at this? Thanks
I am not able to reproduce the issues in the environment below. Please retest with the latest Dockerfile.rocm. I will retest with MI210 as specified by the reporter and report back. I do not expect the results to be any different.
Docker image: built from Dockerfile.rocm in https://github.com/vllm-project/vllm/commit/7ecee3432110bae563c8756a66b54e5f08dc777d torch: 2.5.0.dev20240726+rocm6.1 python3.9 (built in conda env from the image, torch is already installed, ray is not installed) ray 2.34.0 rocm-6.1.2
2024-08-07 23:43:27,414 INFO worker.py:1781 -- Started a local Ray instance.
(GPUActor pid=15430) ##### actor
(GPUActor pid=15430) GPU IDs: ['0']
(GPUActor pid=15430) ROCR_VISIBLE_DEVICES: 0
(GPUActor pid=15430) Number of GPUs: 1
(GPUActor pid=15430) Device 0: _CudaDeviceProperties(name='AMD Instinct MI300X', major=9, minor=4, gcnArchName='gfx942:sramecc+:xnack-', total_memory=196592MB, multi_processor_count=304, uuid=c172637a402d167b)
(GPUActor pid=15430) #####
(gpu_task pid=15553) ##### task
(gpu_task pid=15553) GPU IDs: ['1']
(gpu_task pid=15553) ROCR_VISIBLE_DEVICES: 1
(gpu_task pid=15553) Number of GPUs: 1
(gpu_task pid=15553) Device 0: _CudaDeviceProperties(name='AMD Instinct MI300X', major=9, minor=4, gcnArchName='gfx942:sramecc+:xnack-', total_memory=196592MB, multi_processor_count=304, uuid=84199e04b7dffe69)
(gpu_task pid=15553) #####
Verified on MI210.
Docker image: built from Dockerfile.rocm in https://github.com/vllm-project/vllm/commit/7ecee3432110bae563c8756a66b54e5f08dc777d torch: 2.5.0.dev20240726+rocm6.1 python3.9 (built in conda env from the image, torch is already installed) ray 2.34.0 rocm-6.1.2
root@smc-dh144-dc20-u06:/workspace# ROCR_VISIBLE_DEVICES=0,1 python test.py
2024-08-09 16:54:25,793 INFO worker.py:1781 -- Started a local Ray instance.
(GPUActor pid=45113) ##### actor
(GPUActor pid=45113) GPU IDs: ['0']
(GPUActor pid=45113) ROCR_VISIBLE_DEVICES: 0,1
(GPUActor pid=45113) Number of GPUs: 2
(GPUActor pid=45113) Device 0: _CudaDeviceProperties(name='AMD Instinct MI210', major=9, minor=0, gcnArchName='gfx90a:sramecc+:xnack-', total_memory=65520MB, multi_processor_count=104)
(GPUActor pid=45113) Device 1: _CudaDeviceProperties(name='AMD Instinct MI210', major=9, minor=0, gcnArchName='gfx90a:sramecc+:xnack-', total_memory=65520MB, multi_processor_count=104)
(GPUActor pid=45113) #####
(gpu_task pid=45318) ##### task
(gpu_task pid=45318) GPU IDs: ['1']
(gpu_task pid=45318) ROCR_VISIBLE_DEVICES: 0,1
(gpu_task pid=45318) Number of GPUs: 2
(gpu_task pid=45318) Device 1: _CudaDeviceProperties(name='AMD Instinct MI210', major=9, minor=0, gcnArchName='gfx90a:sramecc+:xnack-', total_memory=65520MB, multi_processor_count=104) [repeated 2x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(gpu_task pid=45318) #####
What happened + What you expected to happen
I'm testing
num_gpus
feature on a 4 MI210 machine. When running test_amd.py withROCR_VISIBLE_DEVICES=0,1
andnum_gpus=1
on task and actors, task/actor expect to each see one 1 GPU. However, the actor getsROCR_VISIBLE_DEVICES=0,1
, torch expect to see 4 devices and crashed trying to get device property.When running same code on Nvidia machine(change
ROCR_VISIBLE_DEVICES
toCUDA_VISIBLE_DEVICES
), the actor and task see 1 GPU as expected.Output from AMD machine:
Output from Nvidia machine
Versions / Dependencies
docker image: https://github.com/vllm-project/vllm/blob/main/Dockerfile.rocm python3.9 (built in conda env from the image, torch and ray is already installed) torch: 2.4.0.dev20240612+rocm6.1 ray: 2.31.0 ROCM: 6.1.2.60102-119~20.04 GPU: MI210
Reproduction script
test_amd.py
To run:
Issue Severity
High: It blocks me from completing my task.