search

szaimen / aio-local-ai

GNU Affero General Public License v3.0
2 stars 1 forks source link

Any possibility to add the CUDA acceleration as an option? #26

Closed corwin007x closed 1 month ago

corwin007x commented 3 months ago

Hi,

would there be any possibility to provide a choice to enable HW acceleration for the container? With each AI query my server changes to a starting jet plane...

I believe it would be this image:

quay.io/go-skynet/local-ai:v2.9.0-cublas-cuda12-ffmpeg

JamborJan commented 3 months ago

According to the local ai docs:

docker run -p 8080:8080 --gpus all --name local-ai -ti localai/localai:latest-aio-gpu-nvidia-cuda-12

I see a CPU spike of 1600% with each prompt :-)

szaimen commented 1 month ago

Hi, this is currently out of scope for this repo. However you could fork this repo and adjust the tag that is used.

You could then do the same with this folder https://github.com/nextcloud/all-in-one/tree/main/community-containers/local-ai and adjust the details in order to make it easy for people to run this project with gpu enabled