search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
26.05k stars 3.82k forks source link

[Bug]: vLLM failing on AWS Inferentia (inf2) #6640

Open EshamAaqib opened 1 month ago

EshamAaqib commented 1 month ago

Your current environment

Collecting environment information...
WARNING 07-22 09:16:28 _custom_ops.py:14] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
PyTorch version: 2.1.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.30.0
Libc version: glibc-2.31

Python version: 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-6.1.92-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
Address sizes:                        48 bits physical, 48 bits virtual
CPU(s):                               96
On-line CPU(s) list:                  0-95
Thread(s) per core:                   2
Core(s) per socket:                   48
Socket(s):                            1
NUMA node(s):                         2
Vendor ID:                            AuthenticAMD
CPU family:                           25
Model:                                1
Model name:                           AMD EPYC 7R13 Processor
Stepping:                             1
CPU MHz:                              2649.998
BogoMIPS:                             5299.99
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            1.5 MiB
L1i cache:                            1.5 MiB
L2 cache:                             24 MiB
L3 cache:                             192 MiB
NUMA node0 CPU(s):                    0-23,48-71
NUMA node1 CPU(s):                    24-47,72-95
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Mitigation; safe RET, no microcode
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid

Versions of relevant libraries:
[pip3] numpy==1.25.2
[pip3] nvidia-nccl-cu12==2.18.1
[pip3] sagemaker_pytorch_inference==2.0.21
[pip3] torch==2.1.2
[pip3] torch-model-archiver==0.11.0
[pip3] torch-neuronx==2.1.2.2.2.0
[pip3] torch-xla==2.1.3
[pip3] torchserve==0.11.0
[pip3] torchvision==0.16.2
[pip3] transformers==4.42.4
[pip3] transformers-neuronx==0.11.351
[pip3] triton==2.1.0
[conda] mkl                       2024.1.0           ha957f24_693    conda-forge
[conda] mkl-include               2024.2.0           ha957f24_663    conda-forge
[conda] numpy                     1.25.2                   pypi_0    pypi
[conda] nvidia-nccl-cu12          2.18.1                   pypi_0    pypi
[conda] sagemaker-pytorch-inference 2.0.21                   pypi_0    pypi
[conda] torch                     2.1.2                    pypi_0    pypi
[conda] torch-model-archiver      0.11.0                   pypi_0    pypi
[conda] torch-neuronx             2.1.2.2.2.0              pypi_0    pypi
[conda] torch-xla                 2.1.3                    pypi_0    pypi
[conda] torchserve                0.11.0                   pypi_0    pypi
[conda] torchvision               0.16.2                   pypi_0    pypi
[conda] transformers              4.42.4                   pypi_0    pypi
[conda] transformers-neuronx      0.11.351                 pypi_0    pypi
[conda] triton                    2.1.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: (0, '+--------+--------+--------+-------------+---------+-------+---------+\n| NEURON | NEURON | NEURON |  CONNECTED  |   PCI   |  PID  | RUNTIME |\n| DEVICE | CORES  | MEMORY |   DEVICES   |   BDF   |       | VERSION |\n+--------+--------+--------+-------------+---------+-------+---------+\n| 0      | 2      | 32 GB  | 12, 3, 4, 1 | 10:1e.0 | 27654 | 2.21.41 |\n| 1      | 2      | 32 GB  | 13, 0, 5, 2 | 20:1e.0 | 27654 | 2.21.41 |\n| 2      | 2      | 32 GB  | 14, 1, 6, 3 | 10:1d.0 | 27654 | 2.21.41 |\n| 3      | 2      | 32 GB  | 15, 2, 7, 0 | 20:1f.0 | 27654 | 2.21.41 |\n+--------+--------+--------+-------------+---------+-------+---------+', '')
vLLM Version: 0.5.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
Could not collect

🐛 Describe the bug

vLLM 0.5.0 is failing on AWS inf2 with the following, I have tried to run the following LLMs but fails with same error 

mistralai/Mistral-7B-Instruct-v0.3
meta-llama/Meta-Llama-3-8B-Instruct

args used to launch vLLM -

            - '--device'
            - neuron
            - '--port'
            - '5000'
            - '--host'
            - 0.0.0.0
            - '--download-dir'
            - /workspace/.cache/hub
            - '--model'
            - meta-llama/Meta-Llama-3-8B
            - '--tensor-parallel-size'
            - '4'
            - '--max-model-len'
            - '8192'
            - '--max-num-seqs'
            - '8'

Error -

INFO 07-22 07:38:20 api_server.py:177] vLLM API server version 0.5.0
INFO 07-22 07:38:20 api_server.py:178] args: Namespace(host='0.0.0.0', port=5000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='meta-llama/Meta-Llama-3-8B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir='/workspace/.cache/hub', load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=8192, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=8, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, device='neuron', image_input_type=None, image_token_id=None, image_input_shape=None, image_feature_size=None, image_processor=None, image_processor_revision=None, disable_image_processor=False, scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model=None, num_speculative_tokens=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, model_loader_extra_config=None, preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
INFO 07-22 07:38:20 config.py:623] Defaulting to use ray for distributed inference
INFO 07-22 07:38:20 llm_engine.py:161] Initializing an LLM engine (v0.5.0) with config: model='meta-llama/Meta-Llama-3-8B', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir='/workspace/.cache/hub', load_format=LoadFormat.AUTO, tensor_parallel_size=4, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=meta-llama/Meta-Llama-3-8B)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
WARNING 07-22 07:38:21 utils.py:465] Pin memory is not supported on Neuron.

Trying to resume download...
WARNING:huggingface_hub.file_download:Error while downloading from https://cdn-lfs-us-1.huggingface.co/repos/ba/83/ba837357c37c4c572f89ebcdffc2867477dbb4768779d332eb303f848c60d0de/4b8fbc5e113f69768dd8de84661ea20af8a32b734a9976144b4236c447b40ccc?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00003-of-00004.safetensors%3B+filename%3D%22model-00003-of-00004.safetensors%22%3B&Expires=1721893130&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMTg5MzEzMH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2JhLzgzL2JhODM3MzU3YzM3YzRjNTcyZjg5ZWJjZGZmYzI4Njc0NzdkYmI0NzY4Nzc5ZDMzMmViMzAzZjg0OGM2MGQwZGUvNGI4ZmJjNWUxMTNmNjk3NjhkZDhkZTg0NjYxZWEyMGFmOGEzMmI3MzRhOTk3NjE0NGI0MjM2YzQ0N2I0MGNjYz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=tBRyl-e7E4csKEY8onfOmx2ddihpcTa7FpzhqFiGo89ONPQ6kvPINFX1d89IHdblcacqtYKvXFvqLvHwyS0KK6tKXETEdJLo8jmU-FJrkCF0Wr7giEtHw3k1YWbqBviJ-5S0l5KEzqUiHuDJ6gc7E-Vd723UkgUEZf49-VLFvwBrkQj3c0oP39RSjkuEO%7EhwPRUwwdZldjfCZ4nhukqNUrpSqw-o%7Efa5-3ahzi1mAGDPmrOXQE2vWkximPnBp53RCH32c4rL7RfLl6QCGY%7EwRg8P%7EkC1UOOCFljxsd7lIo1uAviPFnucgZyEW8snhfxv-xR8winKdpbJUgYYttd8vw__&Key-Pair-Id=K24J24Z295AEI9: HTTPSConnectionPool(host='cdn-lfs-us-1.huggingface.co', port=443): Read timed out.
Trying to resume download...

2024-07-22 07:45:05.000507:  196  INFO ||NEURON_CACHE||: Compile cache path: /var/tmp/neuron-compile-cache
2024-07-22 07:45:05.000510:  196  INFO ||NEURON_CC_WRAPPER||: Call compiler with cmd: neuronx-cc compile --target=trn1 --framework=XLA /tmp/no-user/neuroncc_compile_workdir/8d973c03-0fa5-421d-b280-bf55f7fbec21/model.MODULE_ec9f755d48b417eec39a+2c2d707e.hlo_module.pb --output /tmp/no-user/neuroncc_compile_workdir/8d973c03-0fa5-421d-b280-bf55f7fbec21/model.MODULE_ec9f755d48b417eec39a+2c2d707e.neff --model-type=transformer --auto-cast=none --verbose=35
2024-07-22 07:45:05.000543:  197  INFO ||NEURON_CACHE||: Compile cache path: /var/tmp/neuron-compile-cache
2024-07-22 07:45:05.000546:  197  INFO ||NEURON_CC_WRAPPER||: Call compiler with cmd: neuronx-cc compile --target=trn1 --framework=XLA /tmp/no-user/neuroncc_compile_workdir/4bfa07ef-9266-4224-985d-124eb59dcf15/model.MODULE_8c2a3a7d0bcda5763173+2c2d707e.hlo_module.pb --output /tmp/no-user/neuroncc_compile_workdir/4bfa07ef-9266-4224-985d-124eb59dcf15/model.MODULE_8c2a3a7d0bcda5763173+2c2d707e.neff --model-type=transformer --auto-cast=none --verbose=35
......
Compiler status PASS
2024-07-22 07:45:50.000243:  197  INFO ||NEURON_CACHE||: Compile cache path: /var/tmp/neuron-compile-cache
....
Compiler status PASS
2024-07-22 07:47:24.000349:  196  INFO ||NEURON_CACHE||: Compile cache path: /var/tmp/neuron-compile-cache
2024-Jul-22 07:47:26.0451 1:195 [3] include/socket.h:270 CCOM WARN Skipping IPv6 loopback adddress
2024-Jul-22 07:47:26.0453 1:195 [3] init.cc:125 CCOM WARN NET/Plugin : No plugin found (libnccl-net.so), multi-instance execution will not work
2024-Jul-22 07:47:26.0454 1:195 [3] include/socket.h:270 CCOM WARN Skipping IPv6 loopback adddress
2024-Jul-22 07:47:26.0454 1:194 [2] include/socket.h:270 CCOM WARN Skipping IPv6 loopback adddress
2024-Jul-22 07:47:26.0454 1:192 [0] include/socket.h:270 CCOM WARN Skipping IPv6 loopback adddress
2024-Jul-22 07:47:26.0454 1:193 [1] include/socket.h:270 CCOM WARN Skipping IPv6 loopback adddress
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
WARNING 07-22 07:47:42 serving_chat.py:95] No chat template provided. Chat API will not work.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
WARNING 07-22 07:47:42 serving_embedding.py:141] embedding_mode is False. Embedding API will not work.
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
INFO 07-22 07:47:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:48:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:48:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:48:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:48:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:48:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:48:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:49:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:49:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:49:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:49:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:49:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:49:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:50:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:50:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:50:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:50:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:50:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:50:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:51:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:51:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:51:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:51:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:51:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:51:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:52:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:52:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:52:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:52:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:52:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:52:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:53:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:53:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:53:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:53:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:53:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:53:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:54:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:54:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:54:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:54:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:54:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:54:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:55:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:55:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:55:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:55:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:55:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:55:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:56:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:56:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:56:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:56:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:56:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:56:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:57:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:57:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:57:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:57:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:57:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:57:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:58:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:58:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:58:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:58:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:58:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:58:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:59:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:59:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:59:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:59:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:59:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 07:59:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 08:00:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 08:00:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 08:00:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 08:00:32 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 08:00:42 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 08:00:52 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO:     10.9.66.150:34674 - "GET /version/ HTTP/1.1" 307 Temporary Redirect
INFO:     10.9.66.150:34674 - "GET /version HTTP/1.1" 200 OK
INFO 07-22 08:01:02 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 08:01:12 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 08:01:22 metrics.py:341] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-22 08:01:29 async_llm_engine.py:564] Received request cmpl-8d6bb910c26d4d2ab4fa84e387ed2b80-0: prompt: '<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, best_of=1, presence_penalty=0.14000000000000012, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=-1, min_p=0.0, seed=1, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=['<|end_of_text|>'], stop_token_ids=[128001, 128001], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [128000, 27, 91, 318, 5011, 91, 29, 882, 198, 9906, 27, 91, 318, 6345, 91, 397, 27, 91, 318, 5011, 91, 29, 78191, 198], lora_request: None.
INFO:     10.9.66.150:40154 - "POST /v1/completions HTTP/1.1" 200 OK
ERROR 07-22 08:01:29 async_llm_engine.py:52] Engine background task failed
ERROR 07-22 08:01:29 async_llm_engine.py:52] Traceback (most recent call last):
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 42, in _log_task_completion
ERROR 07-22 08:01:29 async_llm_engine.py:52]     return_value = task.result()
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 532, in run_engine_loop
ERROR 07-22 08:01:29 async_llm_engine.py:52]     has_requests_in_progress = await asyncio.wait_for(
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
ERROR 07-22 08:01:29 async_llm_engine.py:52]     return fut.result()
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 506, in engine_step
ERROR 07-22 08:01:29 async_llm_engine.py:52]     request_outputs = await self.engine.step_async()
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 235, in step_async
ERROR 07-22 08:01:29 async_llm_engine.py:52]     output = await self.model_executor.execute_model_async(
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/workspace/vllm/vllm/executor/neuron_executor.py", line 83, in execute_model_async
ERROR 07-22 08:01:29 async_llm_engine.py:52]     output = await make_async(
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/opt/conda/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR 07-22 08:01:29 async_llm_engine.py:52]     result = self.fn(*self.args, **self.kwargs)
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR 07-22 08:01:29 async_llm_engine.py:52]     return func(*args, **kwargs)
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/workspace/vllm/vllm/worker/neuron_worker.py", line 87, in execute_model
ERROR 07-22 08:01:29 async_llm_engine.py:52]     output = self.model_runner.execute_model(seq_group_metadata_list)
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR 07-22 08:01:29 async_llm_engine.py:52]     return func(*args, **kwargs)
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/workspace/vllm/vllm/worker/neuron_model_runner.py", line 176, in execute_model
ERROR 07-22 08:01:29 async_llm_engine.py:52]     ) = self.prepare_input_tensors(seq_group_metadata_list)
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/workspace/vllm/vllm/worker/neuron_model_runner.py", line 152, in prepare_input_tensors
ERROR 07-22 08:01:29 async_llm_engine.py:52]     seq_lens) = self._prepare_prompt(seq_group_metadata_list)
ERROR 07-22 08:01:29 async_llm_engine.py:52]   File "/workspace/vllm/vllm/worker/neuron_model_runner.py", line 72, in _prepare_prompt
ERROR 07-22 08:01:29 async_llm_engine.py:52]     assert len(block_table) == 1
ERROR 07-22 08:01:29 async_llm_engine.py:52] AssertionError
ERROR:asyncio:Exception in callback functools.partial(<function _log_task_completion at 0x7f37944e0b80>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f37795f7e20>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x7f37944e0b80>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f37795f7e20>>)>
Traceback (most recent call last):
  File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 42, in _log_task_completion
    return_value = task.result()
INFO 07-22 08:01:29 async_llm_engine.py:167] Aborted request cmpl-8d6bb910c26d4d2ab4fa84e387ed2b80-0.
  File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 532, in run_engine_loop
    has_requests_in_progress = await asyncio.wait_for(
  File "/opt/conda/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 506, in engine_step
    request_outputs = await self.engine.step_async()
  File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 235, in step_async
    output = await self.model_executor.execute_model_async(
  File "/workspace/vllm/vllm/executor/neuron_executor.py", line 83, in execute_model_async
    output = await make_async(
  File "/opt/conda/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/workspace/vllm/vllm/worker/neuron_worker.py", line 87, in execute_model
    output = self.model_runner.execute_model(seq_group_metadata_list)
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/workspace/vllm/vllm/worker/neuron_model_runner.py", line 176, in execute_model
    ) = self.prepare_input_tensors(seq_group_metadata_list)
  File "/workspace/vllm/vllm/worker/neuron_model_runner.py", line 152, in prepare_input_tensors
    seq_lens) = self._prepare_prompt(seq_group_metadata_list)
  File "/workspace/vllm/vllm/worker/neuron_model_runner.py", line 72, in _prepare_prompt
    assert len(block_table) == 1
AssertionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
  File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 54, in _log_task_completion
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for theactual cause.
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/opt/conda/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
    await func()
  File "/opt/conda/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
    message = await receive()
  File "/opt/conda/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
    await self.message_event.wait()
  File "/opt/conda/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f36b38d9d50

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/opt/conda/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
    await response(scope, receive, send)
  File "/opt/conda/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
    async with anyio.create_task_group() as task_group:
  File "/opt/conda/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
    raise BaseExceptionGroup(
exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)

In addition to this I tried running vLLM 0.5.2, but ran into the same issue mentioned here https://github.com/vllm-project/vllm/issues/6269#issuecomment-2221751738

liangfu commented 1 month ago

I'm reading the config:

INFO 07-22 07:38:20 llm_engine.py:161] Initializing an LLM engine (v0.5.0) with config: 
model='meta-llama/Meta-Llama-3-8B', speculative_config=None, 
tokenizer='meta-llama/Meta-Llama-3-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, 
tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, 
max_seq_len=8192, 
download_dir='/workspace/.cache/hub', 
load_format=LoadFormat.AUTO, 
tensor_parallel_size=4, 
disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, 
decoding_config=DecodingConfig(guided_decoding_backend='outlines'), 
seed=0, served_model_name=meta-llama/Meta-Llama-3-8B)

I think the error is mostly due to lack of supporting PagedAttention in vLLM for neuron backend at the moment, thus we require max-model-len equal to block-size.

EshamAaqib commented 1 month ago

I'm reading the config:

INFO 07-22 07:38:20 llm_engine.py:161] Initializing an LLM engine (v0.5.0) with config: 
model='meta-llama/Meta-Llama-3-8B', speculative_config=None, 
tokenizer='meta-llama/Meta-Llama-3-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, 
tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, 
max_seq_len=8192, 
download_dir='/workspace/.cache/hub', 
load_format=LoadFormat.AUTO, 
tensor_parallel_size=4, 
disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, 
decoding_config=DecodingConfig(guided_decoding_backend='outlines'), 
seed=0, served_model_name=meta-llama/Meta-Llama-3-8B)

I think the error is mostly due to lack of supporting PagedAttention in vLLM for neuron backend at the moment, thus we require max-model-len equal to block-size.

  • the immediate short-term workaround is to stretch block-size to be equal to max_model_len
  • we are actively investigating the approach to implement PagedAttention, and bring it to vLLM for neuron backend support.

Thanks, I assume its this (https://docs.vllm.ai/en/latest/models/engine_args.html) : 

--block-size
Possible choices: 8, 16, 32

Token block size for contiguous chunks of tokens.

Default: 16

If so, can we set it to the same value as max-model-len or am I missing something ? when I tired it failed with the following 

api_server.py: error: argument --block-size: invalid choice: 8192 (choose from 8, 16, 32)
liangfu commented 1 month ago
EshamAaqib commented 1 month ago
  • short-term solution is to extend the list of block_size options, so that we would be able to set it to desired size (e.g. 8196).
  • mid-term solution is to develop paged attention support on neuron backend.

Thanks @liangfu, extending the block_size seems to work

4096 and 8192 fails with 4 GPUs allocated with memory allocation errors. 2048 or anything lower than that seems to work

2024-Jul-30 11:23:45.105173     1:1     ERROR  TDRV:dmem_alloc_internal                     Failed to alloc DEVICE memory: 1073741824
2024-Jul-30 11:23:45.110278     1:1     ERROR  TDRV:dml_dump                                Wrote nrt memory alloc debug info to /tmp/nrt_mem_log_device_0_66a8cd41.csv
2024-Jul-30 11:23:45.114159     1:1     ERROR  TDRV:log_dev_mem                             Failed to allocate 1.000GB (usage: tensors) on ND 0:NC 0, current utilization:
    * total: 15.813GB
    * tensors: 15.813GB
    * runtime: 1.062KB
    * dma rings: 32.000KB

2024-Jul-30 11:23:45.121919     1:1     ERROR  TDRV:tensor_allocate                         Failed to allocate 1073741824 bytes on DEVICE for tensor UNKNOWN.