search

snexus / llm-search

Querying local documents, powered by LLM
MIT License
479 stars 60 forks source link

Support several CUDA devices #83

Closed vovsgit closed 7 months ago

vovsgit commented 8 months ago

Hello @snexus! Thank for you pretty library!

I have several CUDA devices, but in console I see:

image

How I can set several CUDA devices?

snexus commented 8 months ago

Hi @vovsgit

Embeddings generation doesn't support multiple GPUs at the moment, but for the inference, it should be supported out of the box via underlying backend (e.g. llama cpp)

snexus commented 7 months ago

Please reopen if still having problems (for inference)