Open rchernan-dell opened 5 months ago
same issue here
This must be cupy issue, not Ray. Basically something is not going well with your env + cupy nccl backend.
For the workaround, you can do enforce_eager=True to vllm to workaround this I believe (because it disables cupy backend).
setting enforce_eager=True yields a new bug:
INFO 03-26 13:20:22 llm_engine.py:87] Initializing an LLM engine with config: model='TheBloke/Llama-2-7b-Chat-AWQ', tokenizer='TheBloke/Llama-2-7b-Chat-AWQ', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=2, disable_custom_all_reduce=True, quantization=awq, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, seed=0) Dell-Dev-U:2803813:2803813 [0] NCCL INFO Bootstrap : Using eno1:192.168.1.37<0> Dell-Dev-U:2803813:2803813 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
Dell-Dev-U:2803813:2803813 [0] init.cc:1270 NCCL WARN Invalid config blocking attribute value -2147483648
Traceback (most recent call last):
File "/home/rch/dev/ubiquitous-distributed-ai/ray-vllm-wsl/exampleawq.py", line 16, in
What happened + What you expected to happen
When vLLM distributes the work to run an api_server distributed on 2 machines to Ray, it is expected that Ray will execute the work successfully and run the service. However the following error is thrown:
Versions / Dependencies
2 machines
Both machines:
Reproduction script
Issue Severity
High: It blocks me from completing my task.