Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, MiniCPM, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, GraphRAG, DeepSpeed, vLLM, FastChat, Axolotl, etc.
Apache License 2.0
6.43k
stars
1.24k
forks
source link
vLLM offline_inference.py failed to run on CPU inference #11056
Failed to run python
offline_inference.py
from link for vLLM offline inference on CPU. It seems thatllm.py
has been removed in the previous version.