search

ml-research / LlavaGuard

Apache License 2.0
23 stars 0 forks source link

Issue Running on 8xH100 #4

Closed Karthikreddyk99 closed 4 days ago

Karthikreddyk99 commented 2 months ago

We tried multiple ways of running the model on multi GPUs, but it will only use single GPU.

Can you provide us the correct command to run the model on multi GPUs

lukashelff commented 4 days ago

In SGLang, you can enable multi-GPU tensor parallelism by adding add --tp 2 when starting the server. Please see the documentation. Otherwise you should be able to set device_map='auto' and define the gpus in the environment using CUDA_VISIBLE_DEVICES.