search

EricLBuehler / mistral.rs

Blazingly fast LLM inference.
MIT License
4.4k stars 303 forks source link

Enable multiple CPU from arguments #680

Open lij55 opened 2 months ago

lij55 commented 2 months ago

I have a 32 core AMD CPU and no GP. mistral.rs will only use two of the cores. 2 cores is a bit less. Is it possible to allow to set it through arguments? Ollama will use half of core numbers by default.

Thanks!

EricLBuehler commented 2 months ago

Hi @lij55 can you please let me know what the command you are running is?

lij55 commented 2 months ago

sorry for late reply. it is target/release/mistralrs-server -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3

mert-kurttutan commented 2 months ago

It also depends on which backend you are using. Is it default backend or mkl? (Depending on whether which version of gemm is being utilized)

Each of these have different settings for number of cpus to be used. For instance, mkl is controlled by OMP_NUM_THREADS or MKL_NUM_THREADS environment variable. Irc, candle default backend is controllled by RAYON_NUM_THREADS, try to play with these environment variables to see if there is any change.

But, it is still weird that only 2 cores are being used.

Again, we need to know which cpu backend is used to solve the issue

lij55 commented 2 months ago

it is the default backend. I will try OMP_NUM_THREADS environments. I didn't notice it. Thanks in advance!

EricLBuehler commented 1 month ago

@lij55 did this work?

lij55 commented 1 month ago

sorry for late reply. I tried both OMP_NUM_THREADS and MKL_NUM_THREADS but no effect. It still use 2 cores. Is it related with model ? I used phi3 by target/release/mistralrs-server -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3

Now I'm using ollama.