search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
27.89k stars 4.11k forks source link

[Usage]: There is no response after the "GPU P2P capability or P2P test failed" warning is displayed. What can I do? #4996

Closed wzz981 closed 3 months ago

wzz981 commented 4 months ago

Your current environment

PyTorch version: 2.1.2+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.7 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~16.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.27.7
Libc version: glibc-2.23

Python version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.4.0-210-generic-x86_64-with-glibc2.23
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: Quadro RTX 6000
GPU 1: Quadro RTX 6000
GPU 2: Quadro RTX 6000
GPU 3: Quadro RTX 6000
GPU 4: Quadro RTX 6000
GPU 5: Quadro RTX 6000
GPU 6: Quadro RTX 6000
GPU 7: Quadro RTX 6000

Nvidia driver version: 550.67
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                64
On-line CPU(s) list:   0-63
Thread(s) per core:    2
Core(s) per socket:    16
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
Stepping:              7
CPU MHz:               1599.984
CPU max MHz:           3900.0000
CPU min MHz:           1200.0000
BogoMIPS:              5788.16
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              22528K
NUMA node0 CPU(s):     0-15,32-47
NUMA node1 CPU(s):     16-31,48-63
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.4
[pip3] onnx==1.15.0
[pip3] onnxruntime==1.14.0
[pip3] rapidocr-onnxruntime==1.3.9
[pip3] torch==2.1.2+cu118
[pip3] torchaudio==2.1.2+cu118
[pip3] torchvision==0.16.2+cu118
[pip3] triton==2.1.0
[conda] numpy                     1.24.4                   pypi_0    pypi
[conda] torch                     2.1.2+cu118              pypi_0    pypi
[conda] torchaudio                2.1.2+cu118              pypi_0    pypi
[conda] torchvision               0.16.2+cu118             pypi_0    pypi
[conda] triton                    2.1.0                    pypi_0    pypiROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.4.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PIX     PXB     PXB     PXB     PXB     PXB     PXB     0-15,32-47      0               N/A
GPU1    PIX      X      PXB     PXB     PXB     PXB     PXB     PXB     0-15,32-47      0               N/A
GPU2    PXB     PXB      X      PXB     PXB     PXB     PXB     PXB     0-15,32-47      0               N/A
GPU3    PXB     PXB     PXB      X      PIX     PXB     PXB     PXB     0-15,32-47      0               N/A
GPU4    PXB     PXB     PXB     PIX      X      PXB     PXB     PXB     0-15,32-47      0               N/A
GPU5    PXB     PXB     PXB     PXB     PXB      X      PXB     PXB     0-15,32-47      0               N/A
GPU6    PXB     PXB     PXB     PXB     PXB     PXB      X      PXB     0-15,32-47      0               N/A
GPU7    PXB     PXB     PXB     PXB     PXB     PXB     PXB      X      0-15,32-47      0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

How would you like to use vllm

When I executed export CUDA_VISIBLE_DEVICES=4,5 && python -m vllm.entrypoints.openai.api_server --served-model-name Qwen1.5-4B-Chat --model llm_models/Qwen1.5-4B-Chat --trust-remote-code --port 9741 --dtype float16 --tensor-parallel-size 2 --gpu-memory-utilization 0.55 --max-model-len 5120 --enable-prefix-caching --enforce-eager to test the reasoning, the following appeared:

INFO 05-23 10:32:49 api_server.py:148] vLLM API server version 0.4.0
INFO 05-23 10:32:49 api_server.py:149] args: Namespace(host=None, port=9741, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, served_model_name='Qwen1.5-4B-Chat', lora_modules=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='llm_models/Qwen1.5-4B-Chat', tokenizer=None, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', dtype='float16', kv_cache_dtype='auto', max_model_len=5120, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=True, use_v2_block_manager=False, seed=0, swap_space=4, gpu_memory_utilization=0.55, forced_num_gpu_blocks=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=5, disable_log_stats=False, quantization=None, enforce_eager=True, max_context_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', max_cpu_loras=None, device='auto', image_input_type=None, image_token_id=None, image_input_shape=None, image_feature_size=None, scheduler_delay_factor=0.0, enable_chunked_prefill=False, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
WARNING 05-23 10:32:49 config.py:748] Casting torch.bfloat16 to torch.float16.
2024-05-23 10:32:52,035 INFO worker.py:1724 -- Started a local Ray instance.
INFO 05-23 10:32:57 llm_engine.py:75] Initializing an LLM engine (v0.4.0) with config: model='llm_models/Qwen1.5-4B-Chat', tokenizer='llm_models/Qwen1.5-4B-Chat', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=5120, download_dir=None, load_format=auto, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, seed=0)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 05-23 10:33:08 selector.py:34] Cannot use FlashAttention backend for Volta and Turing GPUs.
INFO 05-23 10:33:08 selector.py:21] Using XFormers backend.
(RayWorkerVllm pid=380843) INFO 05-23 10:33:08 selector.py:34] Cannot use FlashAttention backend for Volta and Turing GPUs.
(RayWorkerVllm pid=380843) INFO 05-23 10:33:08 selector.py:21] Using XFormers backend.
INFO 05-23 10:33:09 pynccl_utils.py:45] vLLM is using nccl==2.9.9
(RayWorkerVllm pid=380843) INFO 05-23 10:33:09 pynccl_utils.py:45] vLLM is using nccl==2.9.9
(RayWorkerVllm pid=380843) WARNING 05-23 10:33:09 custom_all_reduce.py:149] Cannot test GPU P2P because not all GPUs are visible to the current process. This might be the case if 'CUDA_VISIBLE_DEVICES' is set.
(RayWorkerVllm pid=380843) WARNING 05-23 10:33:09 custom_all_reduce.py:45] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.

I use vllm0.3.0 is no problem, but I want to use "--enable-prefix-caching", so try to use vllm0.4.0, but after execution, to "GPU P2P capability or P2P test failed" here has no response, there is no error, wait for more than an hour or so. Is there a component I'm missing or is it a hardware problem?

pisko1979 commented 4 months ago

Hi all

same problem here on k8s. en 15376 --chat-template /chat-templates/mistral.jinja'

thanks regards

DarkLight1337 commented 3 months ago

We have added documentation for this situation in #5430. Please take a look.