search

nomic-ai / gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
https://nomic.ai/gpt4all
MIT License
69.13k stars 7.59k forks source link

n_threads for gpt4all-api #1702

Closed cebtenzzre closed 4 months ago

cebtenzzre commented 9 months ago

Discussed in https://github.com/nomic-ai/gpt4all/discussions/1701

Originally posted by **patyupin** November 30, 2023 I was able to run and use gpt4all-api for my queries, but it always uses 4 CPU cores, no matter what I modify. I was able to run local gpt4all with 24 cores though. How I can configure running docker gpt4all-api container for using all power of the CPUs?
dpsalvatierra commented 9 months ago

Sounds like something you would control at the docker level.

Try this first to see if it will even take your assigned cpus:

docker run --cpus="6" your_image as a test

Then explicitly force more cpus in the docker compose file:

version: '3.8' services: your_service: image: your_image deploy: resources: limits: cpus: '6' # Limit to 6 CPUs

Or via daemon /etc/docker/daemon.json. You can specify the CPUs for Docker to use with the "cpuset-cpus" option. For example, if your machine has 8 CPUs and you want Docker to use all of them, you can set it like this: { "cpuset-cpus": "0-7" }

And restart the docker service

patyupin commented 9 months ago

can you advise me how to run gpt4api directly w/o docker?

dpsalvatierra commented 9 months ago

can you advise me how to run gpt4api directly w/o docker?

You can achieve this by installing FastAPI on your linux box and utilizing the available scripts to configure it. Because every distro is different, I assume that the folks who initially came up with the idea thought it was easier to maintain containerized

cebtenzzre commented 4 months ago

gpt4all-api has been removed, see #2314.