Insufficient VRAM on a single card, how to utilize multiple cards for inference

mbzuai-oryx / Video-ChatGPT

[ACL 2024 🔥] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.

https://mbzuai-oryx.github.io/Video-ChatGPT

Creative Commons Attribution 4.0 International

1.05k stars 92 forks source link

Insufficient VRAM on a single card, how to utilize multiple cards for inference #93

Closed ds-gong closed 3 months ago

ds-gong commented 3 months ago

Hi, I have 2 * 2080TI, each GPU card has 12GB of memory. How can I use the two GPU cards for inference? ![Uploading 111.png…]()

ds-gong commented 3 months ago

111

mmaaz60 commented 3 months ago

Hi @ds-gong,

Thanks for your interest in our work. Unfortunately, currently we only support single GPU inference.