search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
27.88k stars 4.11k forks source link

[Bug]: NCCL gives an error when I use tensor_parallel ：RuntimeError: NCCL error: invalid usage #7201

Closed wlll123456 closed 1 month ago

wlll123456 commented 2 months ago

Your current environment

Collecting environment information... PyTorch version: 2.3.1+cu118 Is debug build: False CUDA used to build PyTorch: 11.8 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.30.1 Libc version: glibc-2.35

Python version: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 11.8.89 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA A40 GPU 1: NVIDIA A40 GPU 2: NVIDIA A40 GPU 3: NVIDIA A40 GPU 4: NVIDIA A40 GPU 5: NVIDIA A40 GPU 6: NVIDIA A40 GPU 7: NVIDIA A40

Nvidia driver version: 520.61.05 cuDNN version: Probably one of the following: /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn.so.8 /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8 /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_adv_train.so.8 /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8 /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8 /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8 /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudnn_ops_train.so.8 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 86 On-line CPU(s) list: 0-85 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz CPU family: 6 Model: 106 Thread(s) per core: 2 Core(s) per socket: 43 Socket(s): 1 Stepping: 6 BogoMIPS: 5187.80 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid md_clear Virtualization: VT-x Hypervisor vendor: KVM Virtualization type: full L1d cache: 2.7 MiB (86 instances) L1i cache: 2.7 MiB (86 instances) L2 cache: 172 MiB (43 instances) L3 cache: 16 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-85 Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown

Versions of relevant libraries: [pip3] numpy==1.26.3 [pip3] nvidia-nccl-cu11==2.20.5 [pip3] pyzmq==26.0.3 [pip3] torch==2.3.1+cu118 [pip3] torchvision==0.18.1+cu118 [pip3] transformers==4.43.1 [pip3] triton==2.3.1 [conda] numpy 1.26.3 pypi_0 pypi [conda] nvidia-nccl-cu11 2.20.5 pypi_0 pypi [conda] pyzmq 26.0.3 pypi_0 pypi [conda] torch 2.3.1+cu118 pypi_0 pypi [conda] torchvision 0.18.1+cu118 pypi_0 pypi [conda] transformers 4.43.1 pypi_0 pypi [conda] triton 2.3.1 pypi_0 pypi ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.5.3.post1 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU0 X PHB PHB PHB PHB PHB PHB PHB 0-85 N/A GPU1 PHB X PHB PHB PHB PHB PHB PHB 0-85 N/A GPU2 PHB PHB X PHB PHB PHB PHB PHB 0-85 N/A GPU3 PHB PHB PHB X PHB PHB PHB PHB 0-85 N/A GPU4 PHB PHB PHB PHB X PHB PHB PHB 0-85 N/A GPU5 PHB PHB PHB PHB PHB X PHB PHB 0-85 N/A GPU6 PHB PHB PHB PHB PHB PHB X PHB 0-85 N/A GPU7 PHB PHB PHB PHB PHB PHB PHB X 0-85 N/A

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug

(vllmenv2) wjc@Z:~/vllm_test/vllm/benchmarks$ python benchmark_throughput.py --model /data/wjc/Qwen/Qwen2-7B-Instruct/ --dataset ShareGPT_V3_unfiltered_cleaned_split.json --gpu_memory_utilization=0.7 --tensor_parallel_size=2 Namespace(backend='vllm', dataset='ShareGPT_V3_unfiltered_cleaned_split.json', input_len=None, output_len=None, model='/data/wjc/Qwen/Qwen2-7B-Instruct/', tokenizer='/data/wjc/Qwen/Qwen2-7B-Instruct/', quantization=None, tensor_parallel_size=2, n=1, use_beam_search=False, num_prompts=1000, seed=0, hf_max_batch_size=None, trust_remote_code=False, max_model_len=None, dtype='auto', gpu_memory_utilization=0.7, enforce_eager=False, kv_cache_dtype='auto', quantization_param_path=None, device='auto', enable_prefix_caching=False, enable_chunked_prefill=False, max_num_batched_tokens=None, download_dir=None, output_json=None, distributed_executor_backend=None, load_format='auto') INFO 08-06 18:38:35 config.py:718] Defaulting to use mp for distributed inference INFO 08-06 18:38:35 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='/data/wjc/Qwen/Qwen2-7B-Instruct/', speculative_config=None, tokenizer='/data/wjc/Qwen/Qwen2-7B-Instruct/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=/data/wjc/Qwen/Qwen2-7B-Instruct/, use_v2_block_manager=False, enable_prefix_caching=False) INFO 08-06 18:38:35 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either:

(VllmWorkerProcess pid=3615391) INFO 08-06 18:38:40 model_runner.py:692] Loading model weights took 7.1216 GB INFO 08-06 18:38:40 model_runner.py:692] Loading model weights took 7.1216 GB INFO 08-06 18:38:48 distributed_gpu_executor.py:56] # GPU blocks: 48320, # CPU blocks: 9362 (VllmWorkerProcess pid=3615391) INFO 08-06 18:38:55 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. (VllmWorkerProcess pid=3615391) INFO 08-06 18:38:55 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage. INFO 08-06 18:38:55 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. INFO 08-06 18:38:55 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage.

iZ1pp06qu51oiorg7t0nrwZ:3615391:3615391 [1] misc/strongstream.cc:53 NCCL WARN NCCL cannot be captured in a graph if either it wasn't built with CUDA runtime >= 11.3 or if the installed CUDA driver < R465.

iZ1pp06qu51oiorg7t0nrwZ:3615176:3615176 [0] misc/strongstream.cc:53 NCCL WARN NCCL cannot be captured in a graph if either it wasn't built with CUDA runtime >= 11.3 or if the installed CUDA driver < R465. (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method initialize_cache: NCCL error: invalid usage (run with NCCL_DEBUG=WARN for details), Traceback (most recent call last): (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] output = executor(*args, kwargs) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/worker/worker.py", line 220, in initialize_cache (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] self._warm_up_model() (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/worker/worker.py", line 236, in _warm_up_model (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] self.model_runner.capture_model(self.gpu_cache) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] return func(args, kwargs) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/worker/model_runner.py", line 1173, in capture_model (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] graph_runner.capture(capture_inputs) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/worker/model_runner.py", line 1411, in capture (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] output_hidden_or_intermediate_states = self.model( (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] return self._call_impl(args, kwargs) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] return forward_call(*args, kwargs) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/model_executor/models/qwen2.py", line 336, in forward (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] hidden_states = self.model(input_ids, positions, kv_caches, (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] return self._call_impl(*args, *kwargs) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] return forward_call(args, kwargs) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/model_executor/models/qwen2.py", line 253, in forward (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] hidden_states = self.embed_tokens(input_ids) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] return self._call_impl(*args, *kwargs) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] return forward_call(args, **kwargs) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/model_executor/layers/vocab_parallel_embedding.py", line 352, in forward (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] output = tensor_model_parallel_all_reduce(output_parallel) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/distributed/communication_op.py", line 11, in tensor_model_parallel_all_reduce (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] return get_tp_group().allreduce(input) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/distributed/parallel_state.py", line 291, in all_reduce (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] pynccl_comm.allreduce(input) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/distributed/device_communicators/pynccl.py", line 118, in all_reduce (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] self.nccl.ncclAllReduce(buffer_type(tensor.data_ptr()), (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/distributed/device_communicators/pynccl_wrapper.py", line 257, in ncclAllReduce (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] self.NCCL_CHECK(self._funcs["ncclAllReduce"](sendbuff, recvbuff, count, (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] File "/home/wjc/vllm_test/vllm/vllm/distributed/device_communicators/pynccl_wrapper.py", line 223, in NCCL_CHECK (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] raise RuntimeError(f"NCCL error: {error_str}") (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] RuntimeError: NCCL error: invalid usage (run with NCCL_DEBUG=WARN for details) (VllmWorkerProcess pid=3615391) ERROR 08-06 18:38:55 multiproc_worker_utils.py:226] rank0: Traceback (most recent call last): rank0: File "/home/wjc/vllm_test/vllm/benchmarks/benchmark_throughput.py", line 440, in

rank0: File "/home/wjc/vllm_test/vllm/benchmarks/benchmark_throughput.py", line 227, in main rank0: elapsed_time = run_vllm( rank0: File "/home/wjc/vllm_test/vllm/benchmarks/benchmark_throughput.py", line 89, in run_vllm rank0: llm = LLM( rank0: File "/home/wjc/vllm_test/vllm/vllm/entrypoints/llm.py", line 155, in init rank0: self.llm_engine = LLMEngine.from_engine_args( rank0: File "/home/wjc/vllm_test/vllm/vllm/engine/llm_engine.py", line 441, in from_engine_args rank0: engine = cls( rank0: File "/home/wjc/vllm_test/vllm/vllm/engine/llm_engine.py", line 265, in init

rank0: File "/home/wjc/vllm_test/vllm/vllm/engine/llm_engine.py", line 377, in _initialize_kv_caches rank0: self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks) rank0: File "/home/wjc/vllm_test/vllm/vllm/executor/distributed_gpu_executor.py", line 62, in initialize_cache

rank0: File "/home/wjc/vllm_test/vllm/vllm/executor/multiproc_gpu_executor.py", line 178, in _run_workers rank0: driver_worker_output = driver_worker_method(*args, **kwargs) rank0: File "/home/wjc/vllm_test/vllm/vllm/worker/worker.py", line 220, in initialize_cache

rank0: File "/home/wjc/vllm_test/vllm/vllm/worker/worker.py", line 236, in _warm_up_model

rank0: File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context rank0: return func(*args, **kwargs) rank0: File "/home/wjc/vllm_test/vllm/vllm/worker/model_runner.py", line 1173, in capture_model

rank0: File "/home/wjc/vllm_test/vllm/vllm/worker/model_runner.py", line 1411, in capture rank0: output_hidden_or_intermediate_states = self.model( rank0: File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl rank0: return self._call_impl(*args, kwargs) rank0: File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl rank0: return forward_call(*args, *kwargs) rank0: File "/home/wjc/vllm_test/vllm/vllm/model_executor/models/qwen2.py", line 336, in forward rank0: hidden_states = self.model(input_ids, positions, kv_caches, rank0: File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl rank0: return self._call_impl(args, kwargs) rank0: File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl rank0: return forward_call(*args, kwargs) rank0: File "/home/wjc/vllm_test/vllm/vllm/model_executor/models/qwen2.py", line 253, in forward rank0: hidden_states = self.embed_tokens(input_ids) rank0: File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl rank0: return self._call_impl(*args, *kwargs) rank0: File "/home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl rank0: return forward_call(args, kwargs) rank0: File "/home/wjc/vllm_test/vllm/vllm/model_executor/layers/vocab_parallel_embedding.py", line 352, in forward rank0: output = tensor_model_parallel_all_reduce(output_parallel) rank0: File "/home/wjc/vllm_test/vllm/vllm/distributed/communication_op.py", line 11, in tensor_model_parallel_all_reduce rank0: return get_tp_group().allreduce(input) rank0: File "/home/wjc/vllm_test/vllm/vllm/distributed/parallel_state.py", line 291, in all_reduce

rank0: File "/home/wjc/vllm_test/vllm/vllm/distributed/device_communicators/pynccl.py", line 118, in all_reduce

rank0: File "/home/wjc/vllm_test/vllm/vllm/distributed/device_communicators/pynccl_wrapper.py", line 257, in ncclAllReduce rank0: self.NCCL_CHECK(self._funcs["ncclAllReduce"](sendbuff, recvbuff, count, rank0: File "/home/wjc/vllm_test/vllm/vllm/distributed/device_communicators/pynccl_wrapper.py", line 223, in NCCL_CHECK rank0: raise RuntimeError(f"NCCL error: {error_str}") rank0: RuntimeError: NCCL error: invalid usage (run with NCCL_DEBUG=WARN for details) /home/wjc/miniconda3/envs/vllmenv2/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

youkaichao commented 2 months ago

WARN NCCL cannot be captured in a graph if either it wasn't built with CUDA runtime >= 11.3 or if the installed CUDA driver < R465

you need to update your driver and runtime.

wlll123456 commented 1 month ago

I solved my problem, thanks

hiyforever commented 1 month ago

I solved my problem, thanks

i meet this too, can u tell how solve it

wlll123456 commented 1 month ago

Compile nccl with your current CUDA environment and let vllm use the new nccl

hiyforever commented 1 month ago

Compile nccl with your current CUDA environment and let vllm use the new nccl

thanks, i will try it

noob-ctrl commented 1 month ago

Compile nccl with your current CUDA environment and let vllm use the new nccl

I also encountered this problem, I don't quite understand how to solve this. Could you please explain in detail how to solve it?

wlll123456 commented 1 month ago

Compile nccl with your current CUDA environment and let vllm use the new nccl

I also encountered this problem, I don't quite understand how to solve this. Could you please explain in detail how to solve it?

I also encountered this problem, I don't quite understand how to solve this. Could you please explain in detail how to solve it? In my server environment, CUDA is 11.8, but my NCCL needs to be built with CUDA 11.8, so I need to recompile NCCL. First install dependencies (CMAKE, GCC) and then download the NCCL source code (git clone https://github.com/NVIDIA/nccl.git & cd nccl) Configure and compile NCCL (make -j src.build CUDA_HOME=/usr/local/cuda-11.8) Install NCCL (sudo make install) Update environment variables to ensure vllm uses the new nccl. 我英文并不好，这是我用谷歌翻译的，我看你头像有中文，如果有其他疑问，你可以把你的微信发给我

zhaotyer commented 1 month ago

Compile nccl with your current CUDA environment and let vllm use the new nccl

I also encountered this problem, I don't quite understand how to solve this. Could you please explain in detail how to solve it?

I also encountered this problem, I don't quite understand how to solve this. Could you please explain in detail how to solve it? In my server environment, CUDA is 11.8, but my NCCL needs to be built with CUDA 11.8, so I need to recompile NCCL. First install dependencies (CMAKE, GCC) and then download the NCCL source code (git clone https://github.com/NVIDIA/nccl.git & cd nccl) Configure and compile NCCL (make -j src.build CUDA_HOME=/usr/local/cuda-11.8) Install NCCL (sudo make install) Update environment variables to ensure vllm uses the new nccl. 我英文并不好，这是我用谷歌翻译的，我看你头像有中文，如果有其他疑问，你可以把你的微信发给我

你好，我的微信是 z849224266，想请教下这个问题