Closed pallavi080596 closed 7 months ago
Set gpu_split_auto
to false and uncomment the gpu_split
line. Please ask questions like this in the discord server. Github issues is moreso for problems with the code itself and suggestions. Closing this issue.
Hi,
I am trying to run tabbyAPI on a Multi-GPU Setup ( AWS EC2 instance -g5.12xlarge). It has 4 GPUs configured. When I am trying to run using the command
docker compose build
, it starts the server but uses a single GPU to handle the load. This is my config.yml fileCan someone help me with running this on multi-GPUs.