search

EricLBuehler / mistral.rs

Blazingly fast LLM inference.
MIT License
3.38k stars 244 forks source link

Enable multiple CPU from arguments #680

Open lij55 opened 4 weeks ago

lij55 commented 4 weeks ago

I have a 32 core AMD CPU and no GP. mistral.rs will only use two of the cores. 2 cores is a bit less. Is it possible to allow to set it through arguments? Ollama will use half of core numbers by default.

Thanks!

EricLBuehler commented 4 weeks ago

Hi @lij55 can you please let me know what the command you are running is?

lij55 commented 3 weeks ago

sorry for late reply. it is target/release/mistralrs-server -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3

mert-kurttutan commented 1 week ago

It also depends on which backend you are using. Is it default backend or mkl? (Depending on whether which version of gemm is being utilized)

Each of these have different settings for number of cpus to be used. For instance, mkl is controlled by OMP_NUM_THREADS or MKL_NUM_THREADS environment variable. Irc, candle default backend is controllled by RAYON_NUM_THREADS, try to play with these environment variables to see if there is any change.

But, it is still weird that only 2 cores are being used.

Again, we need to know which cpu backend is used to solve the issue

lij55 commented 1 week ago

it is the default backend. I will try OMP_NUM_THREADS environments. I didn't notice it. Thanks in advance!