jianyinglangaws
commented
2 months ago I still get this error with the latest https://github.com/vllm-project/vllm/pull/6313 merge.
Traceback (most recent call last):
File "/home/ubuntu/vllm/simple_test.py", line 11, in <module>
llm = LLM(model="facebook/opt-125m",device="neuron")
File "/home/ubuntu/vllm/vllm/entrypoints/llm.py", line 150, in __init__
self.llm_engine = LLMEngine.from_engine_args(
File "/home/ubuntu/vllm/vllm/engine/llm_engine.py", line 421, in from_engine_args
engine = cls(
File "/home/ubuntu/vllm/vllm/engine/llm_engine.py", line 249, in __init__
self.model_executor = executor_class(
File "/home/ubuntu/vllm/vllm/executor/executor_base.py", line 46, in __init__
self._init_executor()
File "/home/ubuntu/vllm/vllm/executor/neuron_executor.py", line 21, in _init_executor
self._init_worker()
File "/home/ubuntu/vllm/vllm/executor/neuron_executor.py", line 26, in _init_worker
self.driver_worker = NeuronWorker(
TypeError: Can't instantiate abstract class NeuronWorker with abstract method execute_worker
WoosukKwon
areanddee
servient-ashwin
liangfu
omrishiv
Your current environment
Collecting environment information... WARNING 07-09 19:46:51 _custom_ops.py:14] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'") PyTorch version: 2.1.2+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.4 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.30.0 Libc version: glibc-2.35
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-5.15.0-1031-aws-x86_64-with-glibc2.35 Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True
CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: AuthenticAMD Model name: AMD EPYC 7R13 Processor CPU family: 25 Model: 1 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 1 BogoMIPS: 5299.99 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid Hypervisor vendor: KVM Virtualization type: full L1d cache: 512 KiB (16 instances) L1i cache: 512 KiB (16 instances) L2 cache: 8 MiB (16 instances) L3 cache: 64 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-7,16-23 NUMA node1 CPU(s): 8-15,24-31 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected
Versions of relevant libraries: [pip3] numpy==1.25.2 [pip3] nvidia-nccl-cu12==2.18.1 [pip3] torch==2.1.2 [pip3] torch-neuronx==2.1.2.2.2.0 [pip3] torch-xla==2.1.3 [pip3] torchvision==0.16.2 [pip3] transformers==4.42.3 [pip3] transformers-neuronx==0.11.351 [pip3] triton==2.1.0 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: (0, 'instance-type: inf2.8xlarge\ninstance-id: i-0d1255be16fb43afc\n+--------+--------+--------+---------+\n| NEURON | NEURON | NEURON | PCI |\n| DEVICE | CORES | MEMORY | BDF |\n+--------+--------+--------+---------+\n| 0 | 2 | 32 GB | 00:1f.0 |\n+--------+--------+--------+---------+', '') vLLM Version: 0.5.1 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: Could not collect
🐛 Describe the bug
Offline Batched Inference Example from your Quickstart page, running on Neuron inf2 (AWS inf2.8xlarge) returns: "TypeError: Can't instantiate abstract class NeuronWorker with abstract method execute_worker". The only change from your example is the example code is adding device="neuron" argument to LLM.
This code should reproduce the error:
from vllm import LLM, SamplingParams
prompts = [ "Hello, my name is", "The president of the United States is", "The capital of France is", "The future of AI is", ] sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="facebook/opt-125m",device="neuron") outputs = llm.generate(prompts, sampling_params)
Print the outputs.
for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")