youkaichao
commented
1 month ago cc @tlrmchlsmth @mgoin for cutlass and fp8
Closed vlsav closed 1 month ago
youkaichao
commented
1 month ago cc @tlrmchlsmth @mgoin for cutlass and fp8
ArlanCooper
commented
1 month ago the same operation, get this error:
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]: return _run_code(code, main_globals, None,
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]: exec(code, run_globals)
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 216, in <module>
[rank0]: engine = AsyncLLMEngine.from_engine_args(
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 431, in from_engine_args
[rank0]: engine = cls(
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 360, in __init__
[rank0]: self.engine = self._init_engine(*args, **kwargs)
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 507, in _init_engine
[rank0]: return engine_class(*args, **kwargs)
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 256, in __init__
[rank0]: self._initialize_kv_caches()
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 353, in _initialize_kv_caches
[rank0]: self.model_executor.determine_num_available_blocks())
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 76, in determine_num_available_blocks
[rank0]: return self.driver_worker.determine_num_available_blocks()
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/vllm/worker/worker.py", line 173, in determine_num_available_blocks
[rank0]: self.model_runner.profile_run()
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 874, in profile_run
[rank0]: self.execute_model(model_input, kv_caches, intermediate_tensors)
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/home/powerop/work/conda/envs/qwen2/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1201, in execute_model
[rank0]: BatchDecodeWithPagedKVCacheWrapper(
[rank0]: TypeError: 'NoneType' object is not callable
wlwqq
commented
1 month ago the same operation, get this error: `TypeError Traceback (most recent call last) Cell In[1], line 5 3 print('param = ',os.environ.get('VLLM_ATTENTION_BACKEND')) 4 from vllm import LLM,SamplingParams ----> 5 llm = LLM(model='/data/wlh/gemma_2_9b_it_prod') 6 prompts = ["Hello , who are you?"] 7 sampling_params = SamplingParams(temperature=0.8, top_p=0.95,tensor_parallel_size=1)
File ~/.local/lib/python3.8/site-packages/vllm/entrypoints/llm.py:150, in LLM.init(self, model, tokenizer, tokenizer_mode, skip_tokenizer_init, trust_remote_code, tensor_parallel_size, dtype, quantization, revision, tokenizer_revision, seed, gpu_memory_utilization, swap_space, enforce_eager, max_context_len_to_capture, max_seq_len_to_capture, disable_custom_all_reduce, kwargs) 128 raise TypeError( 129 "There is no need to pass vision-related arguments anymore.") 130 engine_args = EngineArgs( 131 model=model, 132 tokenizer=tokenizer, (...) 148 kwargs, 149 ) --> 150 self.llm_engine = LLMEngine.from_engine_args( 151 engine_args, usage_context=UsageContext.LLM_CLASS) 152 self.request_counter = Counter()
File ~/.local/lib/python3.8/site-packages/vllm/engine/llm_engine.py:421, in LLMEngine.from_engine_args(cls, engine_args, usage_context) 419 executor_class = GPUExecutor 420 # Create the LLM engine. --> 421 engine = cls( 422 **engine_config.to_dict(), 423 executor_class=executor_class, 424 log_stats=not engine_args.disable_log_stats, 425 usage_context=usage_context, 426 ) 427 return engine
File ~/.local/lib/python3.8/site-packages/vllm/engine/llm_engine.py:263, in LLMEngine.init(self, model_config, cache_config, parallel_config, scheduler_config, device_config, load_config, lora_config, multimodal_config, speculative_config, decoding_config, observability_config, prompt_adapter_config, executor_class, log_stats, usage_context, stat_loggers) 249 self.model_executor = executor_class( 250 model_config=model_config, 251 cache_config=cache_config, (...) 259 prompt_adapter_config=prompt_adapter_config, 260 ) 262 if not self.model_config.embedding_mode: --> 263 self._initialize_kv_caches() 265 # If usage stat is enabled, collect relevant info. 266 if is_usage_stats_enabled():
File ~/.local/lib/python3.8/site-packages/vllm/engine/llm_engine.py:362, in LLMEngine._initialize_kv_caches(self) 355 def _initialize_kv_caches(self) -> None: 356 """Initialize the KV cache in the worker(s). 357 358 The workers will determine the number of blocks in both the GPU cache 359 and the swap CPU cache. 360 """ 361 num_gpu_blocks, num_cpu_blocks = ( --> 362 self.model_executor.determine_num_available_blocks()) 364 if self.cache_config.num_gpu_blocks_override is not None: 365 num_gpu_blocks_override = self.cache_config.num_gpu_blocks_override
File ~/.local/lib/python3.8/site-packages/vllm/executor/gpu_executor.py:78, in GPUExecutor.determine_num_available_blocks(self) 74 def determine_num_available_blocks(self) -> Tuple[int, int]: 75 """Determine the number of available KV blocks by invoking the 76 underlying worker. 77 """ ---> 78 return self.driver_worker.determine_num_available_blocks()
File ~/.local/lib/python3.8/site-packages/torch/utils/_contextlib.py:115, in context_decorator.
File ~/.local/lib/python3.8/site-packages/vllm/worker/worker.py:179, in Worker.determine_num_available_blocks(self) 175 torch.cuda.empty_cache() 177 # Execute a forward pass with dummy inputs to profile the memory usage 178 # of the model. --> 179 self.model_runner.profile_run() 181 # Calculate the number of blocks that can be allocated with the 182 # profiled peak memory. 183 torch.cuda.synchronize()
File ~/.local/lib/python3.8/site-packages/torch/utils/_contextlib.py:115, in context_decorator.
File ~/.local/lib/python3.8/site-packages/vllm/worker/model_runner.py:923, in GPUModelRunnerBase.profile_run(self) 918 if not get_pp_group().is_first_rank: 919 intermediate_tensors = self.model.make_empty_intermediate_tensors( 920 batch_size=batch_size, 921 dtype=self.model_config.dtype, 922 device=self.device) --> 923 self.execute_model(model_input, kv_caches, intermediate_tensors) 924 torch.cuda.synchronize() 925 return
File ~/.local/lib/python3.8/site-packages/torch/utils/_contextlib.py:115, in context_decorator.
File ~/.local/lib/python3.8/site-packages/vllm/worker/model_runner.py:1299, in ModelRunner.execute_model(self, model_input, kv_caches, intermediate_tensors, num_steps) 1293 if self.flashinfer_decode_workspace_buffer is None: 1294 self.flashinfer_decode_workspace_buffer = torch.empty( 1295 FLASHINFER_WORKSPACE_BUFFER_SIZE, 1296 dtype=torch.uint8, 1297 device=self.device) 1298 self.flashinfer_decode_wrapper = \ -> 1299 BatchDecodeWithPagedKVCacheWrapper( 1300 self.flashinfer_decode_workspace_buffer, "NHD") 1301 self.flashinfer_prefill_workspace_buffer = torch.empty( 1302 FLASHINFER_WORKSPACE_BUFFER_SIZE, 1303 dtype=torch.uint8, 1304 device=self.device) 1305 self.flashinfer_prefill_wrapper = \ 1306 BatchPrefillWithPagedKVCacheWrapper( 1307 self.flashinfer_prefill_workspace_buffer, "NHD")
TypeError: 'NoneType' object is not callable `
tlrmchlsmth
commented
1 month ago @vlsav How did you install vLLM? '_OpNamespace' '_C' object has no attribute 'cutlass_scaled_mm_supports_fp8' would show up if there is a version mismatch between the vLLM Python and the compiled binaries for the kernels.
If you installed vLLM from source like:
pip install -e .Then either try rerunning pip install -e . or the following to recompile the kernels and resolve your issue:
python setup.py build_ext --inplace@wlwqq and @ArlanCooper, those are separate errors -- please open up a new issue to keep the conversation focused.
vlsav
commented
1 month ago @tlrmchlsmth By pip from pypi, not from sources:
pip install vllm==0.5.2
tlrmchlsmth
commented
1 month ago Update: https://github.com/vllm-project/vllm/commit/3f3b6b21500bce2061cae33706bd47c8b6663771 made it into 0.5.1, which kind of breaks the version mismatch theory, unfortunately.
tlrmchlsmth
commented
1 month ago @vlsav what do you see when you run the following?
nm <your venv here>/lib/python3.10/site-packages/vllm/_C.abi3.so | grep cutlass_scaled_mm_supports_fp8
vlsav
commented
1 month ago @tlrmchlsmth 00000000002a0b00 T _Z30cutlass_scaled_mm_supports_fp8l but this is right now, after I reverted back to vllm 0.5.1 after pip install vllm==0.5.2: 000000000029cf70 T _Z30cutlass_scaled_mm_supports_fp8l
jueming0312
commented
1 month ago I also encountered the same problem when deploying the gemma-2-9b-it model with VLLM 0.5.2.
twright8
commented
1 month ago +1
tlrmchlsmth
commented
1 month ago 00000000002a0b00 T _Z30cutlass_scaled_mm_supports_fp8l but this is right now, after I reverted back to vllm 0.5.1 after pip install vllm==0.5.2: 000000000029cf70 T _Z30cutlass_scaled_mm_supports_fp8l
This looks right to me -- at least the function is present in the .so file. I'll try to reproduce the problem.
tlrmchlsmth
commented
1 month ago @jueming0312 and @twright8 it'd be helpful if you could share the output of collect_env.py
twright8
commented
1 month ago `PyTorch version: 2.3.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.6 LTS (x86_64) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 Clang version: Could not collect CMake version: version 3.30.0 Libc version: glibc-2.31
Python version: 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:39) [GCC 12.3.0] (64-bit runtime) Python platform: Linux-5.15.154+-x86_64-with-glibc2.31 Is CUDA available: True CUDA runtime version: 12.1.105 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: Tesla T4 GPU 1: Tesla T4
Nvidia driver version: 550.90.07 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True
CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 2 Core(s) per socket: 2 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) CPU @ 2.00GHz Stepping: 3 CPU MHz: 2000.216 BogoMIPS: 4000.43 Hypervisor vendor: KVM Virtualization type: full L1d cache: 64 KiB L1i cache: 64 KiB L2 cache: 2 MiB L3 cache: 38.5 MiB NUMA node0 CPU(s): 0-3 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI SW loop, KVM SW loop Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat md_clear arch_capabilities
Versions of relevant libraries:
[pip3] flake8==7.0.0
[pip3] flashinfer==0.1.0+cu121torch2.3
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] onnx==1.16.1
[pip3] optree==0.12.1
[pip3] pytorch-ignite==0.5.0.post2
[pip3] pytorch-lightning==2.3.3
[pip3] torch==2.3.0
[pip3] torchaudio==2.1.2
[pip3] torchdata==0.7.1
[pip3] torchinfo==1.8.0
[pip3] torchmetrics==1.4.0.post0
[pip3] torchtext==0.16.2
[pip3] torchvision==0.18.0
[pip3] transformers==4.42.3
[pip3] triton==2.3.0
[conda] flashinfer 0.1.0+cu121torch2.3 pypi_0 pypi
[conda] magma-cuda121 2.6.1 1 pytorch
[conda] mkl 2022.1.0 hc2b9512_224
[conda] nccl 2.22.3.1 hbc370b7_0 conda-forge
[conda] numpy 1.26.4 py310hb13e2d6_0 conda-forge
[conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi
[conda] optree 0.12.1 pypi_0 pypi
[conda] pytorch-ignite 0.5.0.post2 pypi_0 pypi
[conda] pytorch-lightning 2.3.3 pypi_0 pypi
[conda] torch 2.3.0 pypi_0 pypi
[conda] torchaudio 2.1.2 pypi_0 pypi
[conda] torchdata 0.7.1 pypi_0 pypi
[conda] torchinfo 1.8.0 pypi_0 pypi
[conda] torchmetrics 1.4.0.post0 pypi_0 pypi
[conda] torchtext 0.16.2 pypi_0 pypi
[conda] torchvision 0.18.0 pypi_0 pypi
[conda] transformers 4.42.3 pypi_0 pypi
[conda] triton 2.3.0 pypi_0 pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PHB 0-3 0 N/A
GPU1 PHB X 0-3 0 N/A
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks`
twright8
commented
1 month ago Think i needed to upgrade flashinfer. But I cant use it as it only supports ampere 8+. There's a solution proposed here: #6173
tlrmchlsmth
commented
1 month ago @twright8 unfortunately, ampere is needed for fp8 quantization support as well.
@ArlanCooper it looks like you need to install flashinfer (you'll want to download a wheel from here https://github.com/flashinfer-ai/flashinfer/releases)
tlrmchlsmth
commented
1 month ago @vlsav I haven't been able to reproduce your problem, either on a H100 or on an L40.
What model exactly are you running with? I.e. what is /models/gemma-2-9b-it? I successfully started a server using neuralmagic/gemma-2-9b-it-FP8 but when I tried google/gemma-2b-it, I ran into https://github.com/flashinfer-ai/flashinfer/issues/362
vlsav
commented
1 month ago @tlrmchlsmth should it help me if I will install newer version of flashinfer with vllm 0.5.2? I used google/gemma-2b-it
vlsav
commented
1 month ago @tlrmchlsmth now tried to launch neuralmagic/gemma-2-9b-it-FP8 with vllm==0.5.2
[rank0]: Traceback (most recent call last):
[rank0]: File "/opt/llm/miniconda3/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]: return _run_code(code, main_globals, None,
[rank0]: File "/opt/llm/miniconda3/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]: exec(code, run_globals)
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 282, in <module>
[rank0]: run_server(args)
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 224, in run_server
[rank0]: if llm_engine is not None else AsyncLLMEngine.from_engine_args(
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 444, in from_engine_args
[rank0]: engine = cls(
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 373, in __init__
[rank0]: self.engine = self._init_engine(*args, **kwargs)
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 520, in _init_engine
[rank0]: return engine_class(*args, **kwargs)
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 249, in __init__
[rank0]: self.model_executor = executor_class(
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 150, in __init__
[rank0]: super().__init__(model_config, cache_config, parallel_config,
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 46, in __init__
[rank0]: self._init_executor()
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 25, in _init_executor
[rank0]: self.driver_worker.load_model()
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/worker/worker.py", line 139, in load_model
[rank0]: self.model_runner.load_model()
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 256, in load_model
[rank0]: self.model = get_model(model_config=self.model_config,
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 21, in get_model
[rank0]: return loader.load_model(model_config=model_config,
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 267, in load_model
[rank0]: model = _initialize_model(model_config, self.load_config,
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 104, in _initialize_model
[rank0]: return model_class(config=model_config.hf_config,
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py", line 323, in __init__
[rank0]: self.model = Gemma2Model(config, cache_config, quant_config)
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py", line 251, in __init__
[rank0]: self.layers = nn.ModuleList([
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py", line 252, in <listcomp>
[rank0]: Gemma2DecoderLayer(layer_idx, config, cache_config, quant_config)
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py", line 178, in __init__
[rank0]: self.self_attn = Gemma2Attention(
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py", line 115, in __init__
[rank0]: self.qkv_proj = QKVParallelLinear(
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 517, in __init__
[rank0]: super().__init__(input_size=input_size,
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 244, in __init__
[rank0]: super().__init__(input_size, output_size, skip_bias_add, params_dtype,
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 158, in __init__
[rank0]: self.quant_method = quant_config.get_quant_method(self)
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/fp8.py", line 74, in get_quant_method
[rank0]: return Fp8LinearMethod(self)
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/fp8.py", line 105, in __init__
[rank0]: self.cutlass_fp8_supported = cutlass_fp8_supported()
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/utils/w8a8_utils.py", line 15, in cutlass_fp8_supported
[rank0]: return ops.cutlass_scaled_mm_supports_fp8(capability)
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/vllm/_custom_ops.py", line 220, in cutlass_scaled_mm_supports_fp8
[rank0]: return torch.ops._C.cutlass_scaled_mm_supports_fp8(cuda_device_capability)
[rank0]: File "/home/testvllm/.local/lib/python3.10/site-packages/torch/_ops.py", line 921, in __getattr__
[rank0]: raise AttributeError(
[rank0]: AttributeError: '_OpNamespace' '_C' object has no attribute 'cutlass_scaled_mm_supports_fp8'
@tlrmchlsmth should it help me if I will install newer version of flashinfer with vllm 0.5.2? I used google/gemma-2b-it
I don't think that will resolve your issue. This is a more of a linker issue with a C++ function that we compile and ship with the vLLM wheel file. One thing to check is to make sure there isn't a stale _C.abi3.so somewhere on your LD_LIBRARY_PATH.
tlrmchlsmth
commented
1 month ago It also occurs to me that the name _C.abi3.so is extremely generic and we may be having a name collision with some other project
vlsav
commented
1 month ago @tlrmchlsmth should it help me if I will install newer version of flashinfer with vllm 0.5.2? I used google/gemma-2b-it
I don't think that will resolve your issue. This is a more of a linker issue with a C++ function that we compile and ship with the vLLM wheel file. One thing to check is to make sure there isn't a stale
_C.abi3.sosomewhere on yourLD_LIBRARY_PATH.
There is only one _C.abi3.so in ~/.local/lib/python3.10/site-packages/vllm
vlsav
commented
1 month ago @tlrmchlsmth I also found that one in vllm output: WARNING 07-17 20:03:10 _custom_ops.py:14] Failed to import from vllm._C with ImportError("/lib64/libc.so.6: version `GLIBC_2.32' not found (required by /home/testvllm/.local/lib/python3.10/site-packages/vllm/_C.abi3.so)") so it correlates with #6464
tlrmchlsmth
commented
1 month ago @vlsav That is actually your problem: /lib64/libc.so.6: version `GLIBC_2.32' not found, is preventing _C.abi3.so from being loaded.
I'll look into this -- looks like we'd need to build our .so files on an OS with an earlier version of GLIBC, since you're on 2.28
function2-llx
commented
1 month ago FYI, with vllm 0.5.2, I get this warning on Ubuntu 20.04, but on Ubuntu 22.04 it works fine.
@tlrmchlsmth I also found that one in vllm output: WARNING 07-17 20:03:10 _custom_ops.py:14] Failed to import from vllm._C with ImportError("/lib64/libc.so.6: version `GLIBC_2.32' not found (required by /home/testvllm/.local/lib/python3.10/site-packages/vllm/_C.abi3.so)") so it correlates with #6464
vlsav
commented
1 month ago Thanks. Not sure if I will be able to do system upgrade
FYI, with vllm 0.5.2, I get this warning on Ubuntu 20.04, but on Ubuntu 22.04 it works fine.
@tlrmchlsmth I also found that one in vllm output: WARNING 07-17 20:03:10 _custom_ops.py:14] Failed to import from vllm._C with ImportError("/lib64/libc.so.6: version `GLIBC_2.32' not found (required by /home/testvllm/.local/lib/python3.10/site-packages/vllm/_C.abi3.so)") so it correlates with #6464
tlrmchlsmth
commented
1 month ago As a workaround, you can try installing from the wheel files on Github https://github.com/vllm-project/vllm/releases/tag/v0.5.2, which were built on an older OS (Ubuntu 20.04). I think that should work for you.
vlsav
commented
1 month ago As a workaround, you can try installing from the wheel files on Github https://github.com/vllm-project/vllm/releases/tag/v0.5.2, which were built on an older OS (Ubuntu 20.04). I think that should work for you.
Thanks. It works, both for neuralmagic/gemma-2-9b-it-FP8 and google/gemma-2-9b-it
tlrmchlsmth
commented
1 month ago This issue should be fixed for most people in 0.5.3 and later, now that we are building on Ubuntu 20.04. I think we can go ahead and close this one.
vlsav
commented
1 month ago Thanks. So far no issues with 0.5.3
Your current environment
🐛 Describe the bug
Successfully launched gemma-2-9b-it with vlmm 0.5.1. Following script was used export VLLM_ATTENTION_BACKEND=FLASHINFER python -m vllm.entrypoints.openai.api_server --port=8080 --host=0.0.0.0 --model /models/gemma-2-9b-it --quantization fp8 --enforce-eager --seed 1234 --served-model-name gemma-2-9b no issues (except sliding window warning and capping the max length to the sliding window size (4096). Same script after installing vllm 0.5.2 gives error message: