mgoin
commented
1 month ago @ilya-lavrenov @helena-intel can you look into this?
Closed HPUedCSLearner closed 1 month ago
mgoin
commented
1 month ago @ilya-lavrenov @helena-intel can you look into this?
helena-intel
commented
1 month ago Hi @HPUedCSLearner, thanks again for the great bug report! Local models should definitely work. Could you try if it works if you export the model with task text-generation-with-past instead of text-generation? See https://huggingface.co/docs/optimum/main/intel/openvino/export#decoder-models . The command is then::
optimum-cli export openvino -m Qwen1.5-4B-Chat --task text-generation-with-past --weight-format int4 Qwen1.5-4B-Chat-optimum-int4This is also the default for CausalLM models, so omitting the --task parameter should also work if you load a model from the Hugging Face hub.
HPUedCSLearner
commented
1 month ago --weight-format int4
thanks a lot, it works by set the --task --task text-generation-with-past
thanks again!
(vllm-openvino) yongshuai_wang@cpu-10-48-1-249:~/models$ VLLM_OPENVINO_KVCACHE_SPACE=30 \
LLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8 \
VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON \
python3 -m vllm.entrypoints.openai.api_server \
--model /home/yongshuai_wang/models/Qwen1.5-4B-Chat-optimum-int4 \
--port 10003
WARNING 07-17 09:39:30 _custom_ops.py:14] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
/home/yongshuai_wang/miniconda3/envs/vllm-openvino/lib/python3.10/site-packages/vllm/usage/usage_lib.py:19: RuntimeWarning: Failed to read commit hash:
No module named 'vllm.commit_id'
from vllm.version import __version__ as VLLM_VERSION
INFO 07-17 09:39:33 api_server.py:212] vLLM API server version 0.5.2
INFO 07-17 09:39:33 api_server.py:213] args: Namespace(host=None, port=10003, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='/home/yongshuai_wang/models/Qwen1.5-4B-Chat-optimum-int4', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, model_loader_extra_config=None, preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
INFO 07-17 09:39:33 config.py:1374] Downcasting torch.float32 to torch.float16.
INFO 07-17 09:39:33 llm_engine.py:174] Initializing an LLM engine (v0.5.2) with config: model='/home/yongshuai_wang/models/Qwen1.5-4B-Chat-optimum-int4', speculative_config=None, tokenizer='/home/yongshuai_wang/models/Qwen1.5-4B-Chat-optimum-int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=/home/yongshuai_wang/models/Qwen1.5-4B-Chat-optimum-int4, use_v2_block_manager=False, enable_prefix_caching=False)
WARNING 07-17 09:39:34 openvino_executor.py:132] Only float32 dtype is supported on OpenVINO, casting from torch.float16.
WARNING 07-17 09:39:34 openvino_executor.py:137] CUDA graph is not supported on OpenVINO backend, fallback to the eager mode.
INFO 07-17 09:39:34 openvino_executor.py:159] OpenVINO optimal block size is 32, overriding currently set 16
INFO 07-17 09:39:36 selector.py:121] Cannot use _Backend.FLASH_ATTN backend on OpenVINO.
INFO 07-17 09:39:36 selector.py:69] Using OpenVINO Attention backend.
WARNING 07-17 09:39:36 openvino.py:130] OpenVINO IR is available for provided model id /home/yongshuai_wang/models/Qwen1.5-4B-Chat-optimum-int4. This IR will be used for inference as-is, all possible options that may affect model conversion are ignored.
INFO:nncf:Statistics of the bitwidth distribution:
ββββββββββββββββββ―ββββββββββββββββββββββββββββββ―βββββββββββββββββββββββββββββββββββββββββ
β Num bits (N) β % all parameters (layers) β % ratio-defining parameters (layers) β
ββββββββββββββββββΏββββββββββββββββββββββββββββββΏβββββββββββββββββββββββββββββββββββββββββ₯
ββββββββββββββββββ·ββββββββββββββββββββββββββββββ·βββββββββββββββββββββββββββββββββββββββββ
INFO 07-17 09:39:40 openvino_executor.py:72] # CPU blocks: 2457
INFO 07-17 09:39:51 serving_chat.py:94] Using default chat template:
INFO 07-17 09:39:51 serving_chat.py:94] {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
INFO 07-17 09:39:51 serving_chat.py:94] You are a helpful assistant.<|im_end|>
INFO 07-17 09:39:51 serving_chat.py:94] ' }}{% endif %}{{'<|im_start|>' + message['role'] + '
INFO 07-17 09:39:51 serving_chat.py:94] ' + message['content'] + '<|im_end|>' + '
INFO 07-17 09:39:51 serving_chat.py:94] '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
INFO 07-17 09:39:51 serving_chat.py:94] ' }}{% endif %}
WARNING 07-17 09:39:51 serving_embedding.py:141] embedding_mode is False. Embedding API will not work.
INFO 07-17 09:39:51 api_server.py:257] Available routes are:
INFO 07-17 09:39:51 api_server.py:262] Route: /openapi.json, Methods: HEAD, GET
INFO 07-17 09:39:51 api_server.py:262] Route: /docs, Methods: HEAD, GET
INFO 07-17 09:39:51 api_server.py:262] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 07-17 09:39:51 api_server.py:262] Route: /redoc, Methods: HEAD, GET
INFO 07-17 09:39:51 api_server.py:262] Route: /health, Methods: GET
INFO 07-17 09:39:51 api_server.py:262] Route: /tokenize, Methods: POST
INFO 07-17 09:39:51 api_server.py:262] Route: /detokenize, Methods: POST
INFO 07-17 09:39:51 api_server.py:262] Route: /v1/models, Methods: GET
INFO 07-17 09:39:51 api_server.py:262] Route: /version, Methods: GET
INFO 07-17 09:39:51 api_server.py:262] Route: /v1/chat/completions, Methods: POST
INFO 07-17 09:39:51 api_server.py:262] Route: /v1/completions, Methods: POST
INFO 07-17 09:39:51 api_server.py:262] Route: /v1/embeddings, Methods: POST
INFO: Started server process [242065]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:10003 (Press CTRL+C to quit)
INFO 07-17 09:40:01 metrics.py:295] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-17 09:40:11 metrics.py:295] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.I have another question, I would be grateful if someone could tell me.
Can anyone tell me why I got the warning that WARNING 07-17 09:39:30 _custom_ops.py:14] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'") /home/yongshuai_wang/miniconda3/envs/vllm-openvino/lib/python3.10/site-packages/vllm/usage/usage_lib.py:19: RuntimeWarning: Failed to read commit hash: No module named 'vllm.commit_id'
Is it ok if I ignore the WARNING when I use vllm openvino backend?
HPUedCSLearner
commented
1 month ago ith task
text-generation-with-pastinstead oftext-generation? See I have another question about cpu core load, I would be grateful if someone could tell me.
this is my computer numactl info:
NUMA:
NUMA node(s): 4
NUMA node0 CPU(s): 0-15,64-79
NUMA node1 CPU(s): 16-31,80-95
NUMA node2 CPU(s): 32-47,96-111
NUMA node3 CPU(s): 48-63,112-127numactl -m 0 -C 0-15,64-79, but only half cores have work loadVLLM_OPENVINO_KVCACHE_SPACE=100 \
LLM_OPENVINO_CPU_KV_CACHE_PRECISION=u8 \
VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS=ON \
numactl -m 0 -C 0-15,64-79 \
python3 -m vllm.entrypoints.openai.api_server \
--model /home/yongshuai_wang/models/Qwen1.5-4B-Chat-optimum-int4 \
--port 10003@HPUedCSLearner I'm glad you got it to work! We should mention this in the OpenVINO vLLM installation documentation so other people don't run into the same issue. We'll fix that (maybe together with some other updates in the near future).
packages/vllm/usage/usage_lib.py:19: RuntimeWarning: Failed to read commit hash: No module named 'vllm.commit_id'Is it ok if I ignore the WARNING when I use vllm openvino backend?
It is a warning for usage reporting, it's safe to ignore. But if this warning is caused by the OpenVINO backend, we should look into it.
Is there any way, that I can make all the cores I set work?
OpenVINO uses the same number of threads as the number of physical cores it has available. I'm assuming your system uses sub NUMA clustering (SNC). I don't have access to a system with that at the moment and have no experience with that with vLLM. If you have sysadmin access to this system you could consider disabling SNC, so you'll get more cores per NUMA node. Also note that for now vLLM with OpenVINO only works on a single socket.
HPUedCSLearner
commented
1 month ago @HPUedCSLearner I'm glad you got it to work! We should mention this in the OpenVINO vLLM installation documentation so other people don't run into the same issue. We'll fix that (maybe together with some other updates in the near future).
packages/vllm/usage/usage_lib.py:19: RuntimeWarning: Failed to read commit hash: No module named 'vllm.commit_id'Is it ok if I ignore the WARNING when I use vllm openvino backend?It is a warning for usage reporting, it's safe to ignore. But if this warning is caused by the OpenVINO backend, we should look into it.
Is there any way, that I can make all the cores I set work?
OpenVINO uses the same number of threads as the number of physical cores it has available. I'm assuming your system uses sub NUMA clustering (SNC). I don't have access to a system with that at the moment and have no experience with that with vLLM. If you have sysadmin access to this system you could consider disabling SNC, so you'll get more cores per NUMA node. Also note that for now vLLM with OpenVINO only works on a single socket.
Thank you very much for your answer. I personally think that if OpenVINO-vLLM can run in a hyperthreading environment, reasoning should be able to achieve a higher throughput, because I have tested OpenVINO model_server, which can run the entire core in a hyperthreading environment; In addition, I have also tested IPEX-LLM, and its throughput is not as high as OpenVINO-vLLM; More over, if OpenVINO-vLLM can perform cross-Socket reasoning, then it can reason about large models with larger parameters, like 32B 72B LLM; Hope OpenVINO-vLLM can add these two features soon.
liuzhipengchd
commented
6 days ago @HPUedCSLearner HiοΌIf VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS is not enabled, will the inference speed be very slow? What about the concurrency?
ilya-lavrenov
commented
5 days ago @HPUedCSLearner HiοΌIf VLLM_OPENVINO_ENABLE_QUANTIZED_WEIGHTS is not enabled, will the inference speed be very slow? What about the concurrency?
You can see the difference here https://github.com/vllm-project/vllm/pull/5379#issue-2344007685 if you open spoilers with plots. You can see that FP16 model performs even better.
Generally, int8 should have better performance, but currently, we have extra optimizations for FP16 weights and that is why it has better performance. We are in progress of dynamic quantization enabling using AMX which will fully utilize compressed weights.
BarrinXu
commented
1 hour ago Hi @ilya-lavrenov , I've encountered a strange issue when using local openvino model. When sending a long prompt (1024 tokens) to a fp16 format ov model, the vllm would crash without any error log.
optimum-cli export openvino --model Llama-2-7b-chat-hf --weight-format fp16 --task text-generation-with-past Llama-2-7b-chat-ov-fp16python -m vllm.entrypoints.openai.api_server --model /home/chxu/llama_env/7b/Llama-2-7b-chat-ov-fp16 --port 8000Output:
INFO 08-27 20:17:52 importing.py:10] Triton not installed; certain GPU-related functions will be not be available.
WARNING 08-27 20:17:52 _custom_ops.py:17] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
INFO 08-27 20:17:54 api_server.py:408] vLLM API server version 0.5.4
INFO 08-27 20:17:54 api_server.py:409] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, model='/home/chxu/llama_env/7b/Llama-2-7b-chat-ov-fp16', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
INFO 08-27 20:17:54 config.py:1552] Downcasting torch.float32 to torch.float16.
INFO 08-27 20:17:54 api_server.py:135] Multiprocessing frontend to use ipc:///tmp/a3d06905-cb3e-4942-a44c-dc4fb841a41d for RPC Path.
INFO 08-27 20:17:54 api_server.py:146] Started engine process with PID 858824
INFO 08-27 20:17:55 importing.py:10] Triton not installed; certain GPU-related functions will be not be available.
WARNING 08-27 20:17:55 _custom_ops.py:17] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
INFO 08-27 20:17:56 config.py:1552] Downcasting torch.float32 to torch.float16.
INFO 08-27 20:17:56 llm_engine.py:184] Initializing an LLM engine (v0.5.4) with config: model='/home/chxu/llama_env/7b/Llama-2-7b-chat-ov-fp16', speculative_config=None, tokenizer='/home/chxu/llama_env/7b/Llama-2-7b-chat-ov-fp16', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/home/chxu/llama_env/7b/Llama-2-7b-chat-ov-fp16, use_v2_block_manager=False, enable_prefix_caching=False)
WARNING 08-27 20:17:56 openvino_executor.py:133] Only float32 dtype is supported on OpenVINO, casting from torch.float16.
WARNING 08-27 20:17:56 openvino_executor.py:138] CUDA graph is not supported on OpenVINO backend, fallback to the eager mode.
INFO 08-27 20:17:56 openvino_executor.py:160] OpenVINO optimal block size is 32, overriding currently set 16
WARNING 08-27 20:17:56 openvino_executor.py:169] Environment variable VLLM_OPENVINO_KVCACHE_SPACE (GB) for OpenVINO backend is not set, using 4 by default.
INFO 08-27 20:17:58 selector.py:188] Cannot use _Backend.FLASH_ATTN backend on OpenVINO.
INFO 08-27 20:17:58 selector.py:132] Using OpenVINO Attention backend.
WARNING 08-27 20:17:58 openvino.py:130] OpenVINO IR is available for provided model id /home/chxu/llama_env/7b/Llama-2-7b-chat-ov-fp16. This IR will be used for inference as-is, all possible options that may affect model conversion are ignored.
INFO 08-27 20:17:59 openvino_executor.py:73] # CPU blocks: 256
INFO 08-27 20:18:01 api_server.py:197] vLLM to use /tmp/tmp7e3febh1 as PROMETHEUS_MULTIPROC_DIR
WARNING 08-27 20:18:01 serving_embedding.py:188] embedding_mode is False. Embedding API will not work.
INFO 08-27 20:18:01 launcher.py:20] Available routes are:
INFO 08-27 20:18:01 launcher.py:28] Route: /openapi.json, Methods: HEAD, GET
INFO 08-27 20:18:01 launcher.py:28] Route: /docs, Methods: HEAD, GET
INFO 08-27 20:18:01 launcher.py:28] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 08-27 20:18:01 launcher.py:28] Route: /redoc, Methods: HEAD, GET
INFO 08-27 20:18:01 launcher.py:28] Route: /health, Methods: GET
INFO 08-27 20:18:01 launcher.py:28] Route: /tokenize, Methods: POST
INFO 08-27 20:18:01 launcher.py:28] Route: /detokenize, Methods: POST
INFO 08-27 20:18:01 launcher.py:28] Route: /v1/models, Methods: GET
INFO 08-27 20:18:01 launcher.py:28] Route: /version, Methods: GET
INFO 08-27 20:18:01 launcher.py:28] Route: /v1/chat/completions, Methods: POST
INFO 08-27 20:18:01 launcher.py:28] Route: /v1/completions, Methods: POST
INFO 08-27 20:18:01 launcher.py:28] Route: /v1/embeddings, Methods: POST
INFO: Started server process [858756]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)/v1/completions endpoint
This 32-prompt-token json payload (in python) is OK to run:
payload = {
'model': '/home/chxu/llama_env/7b/Llama-2-7b-chat-ov-fp16',
'prompt': [i for i in range(1, 33)], # Make a 32-token prompt.
'min_tokens': 16,
'max_tokens': 16,
'stream': True
}However, this 1024-prompt-token json payload failed and cause the vllm backend to crash without any logging output:
payload = {
'model': '/home/chxu/llama_env/7b/Llama-2-7b-chat-ov-fp16',
'prompt': [i for i in range(1, 1025)], # Make a 1024-token prompt.
'min_tokens': 16,
'max_tokens': 16,
'stream': True
}However, this 1024-prompt-token json payload can successfully be served, if I directly use the huggingface-format model path and let openvino to convert the model after vlllm start.
That is, we directly run python -m vllm.entrypoints.openai.api_server --model /home/chxu/llama_env/7b/Llama-2-7b-chat-hf --port 8000, and wait for the vllm to convert the model to ov format, and then we can successfully post the json payload.
Here's my environment info.
INFO 08-27 20:38:35 importing.py:10] Triton not installed; certain GPU-related functions will be not be available.
WARNING 08-27 20:38:35 _custom_ops.py:17] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
PyTorch version: 2.4.0+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: Ubuntu 24.04 LTS (x86_64)
GCC version: (Ubuntu 13.2.0-23ubuntu4) 13.2.0
Clang version: Could not collect
CMake version: version 3.30.2
Libc version: glibc-2.39
Python version: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.39
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Vendor ID: GenuineIntel
Model name: INTEL(R) XEON(R) PLATINUM 8562Y+
CPU family: 6
Model: 207
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
Stepping: 2
CPU(s) scaling MHz: 20%
CPU max MHz: 4100.0000
CPU min MHz: 800.0000
BogoMIPS: 5600.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req hfi vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 3 MiB (64 instances)
L1i cache: 2 MiB (64 instances)
L2 cache: 128 MiB (64 instances)
L3 cache: 120 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111,113,115,117,119,121,123,125,127
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] onnx==1.16.2
[pip3] pyzmq==26.1.1
[pip3] torch==2.4.0+cpu
[pip3] transformers==4.43.4
[conda] numpy 1.26.4 pypi_0 pypi
[conda] pyzmq 26.1.1 pypi_0 pypi
[conda] torch 2.4.0+cpu pypi_0 pypi
[conda] transformers 4.43.4 pypi_0 pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.4@baaedfdb2d3f1d70b7dbcde08b083abfe6017a92
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
Could not collectMany thanks in advance.
Your current environment
π Describe the bug
1γbug description
It is normal to let vllm-openvino convert openvino IR at runtime; However, manually converting the model to openvino IR will result in an error
use_cacheXXXX2γmanualy convert module to OpenVINO IRοΌand run ,get error:
convert commad
convert OpenVION IR logs
run command
use manuly convert model : /home/yongshuai_wang/models/Qwen1.5-4B-Chat-optimum-int4
get use_cache error
3γhowever, directly run vllm openvion with original modle Qwen1.5-4B-Chat οΌ is OK:
run log