Open lij55 opened 4 weeks ago
Hi @lij55 can you please let me know what the command you are running is?
sorry for late reply.
it is target/release/mistralrs-server -i plain -m microsoft/Phi-3-mini-128k-instruct -a phi3
It also depends on which backend you are using. Is it default backend or mkl? (Depending on whether which version of gemm is being utilized)
Each of these have different settings for number of cpus to be used. For instance, mkl is controlled by OMP_NUM_THREADS or MKL_NUM_THREADS environment variable. Irc, candle default backend is controllled by RAYON_NUM_THREADS, try to play with these environment variables to see if there is any change.
But, it is still weird that only 2 cores are being used.
Again, we need to know which cpu backend is used to solve the issue
it is the default backend. I will try OMP_NUM_THREADS environments. I didn't notice it. Thanks in advance!
I have a 32 core AMD CPU and no GP. mistral.rs will only use two of the cores. 2 cores is a bit less. Is it possible to allow to set it through arguments? Ollama will use half of core numbers by default.
Thanks!