Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, MiniCPM, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, GraphRAG, DeepSpeed, vLLM, FastChat, Axolotl, etc.
Apache License 2.0
6.43k
stars
1.24k
forks
source link
Is there a way to run ollama with IPEX-LLM on CPU #11250
I want to run ollama with IPEM-LLM on a machine with 4 Intel Xeon CPU E7-4830 v3 processors and 256GB of memory. The operating system is Ubuntu 24.04. I followed the steps in the official tutorial as follows:
Hi @reeingal, ollama with IPEX-LLM does not support running on a pure CPU platform, as we haven't optimized ollama for CPU. You may switch to a GPU device to enable IPEX-LLM optimizations.
I want to run ollama with IPEM-LLM on a machine with 4 Intel Xeon CPU E7-4830 v3 processors and 256GB of memory. The operating system is Ubuntu 24.04. I followed the steps in the official tutorial as follows:
init-ollama
:But when I selected the model on open-webui and sent the question, I received a response with error code
500
.I checked the console, and the last output was as follows:
I couldn't find a method to run ollama with IPEX-LLM on a CPU in the official documentation. I hope someone can point out the problem for me.