openai_api_demo.py on 2 nvidia 3090 GPU

THUDM / CogVLM2

GPT4V-level open-source multi-modal model based on Llama3-8B

Apache License 2.0

1.41k stars 77 forks source link

openai_api_demo.py on 2 nvidia 3090 GPU #116

Open arit2 opened 6 days ago

arit2 commented 6 days ago

Hi Great project. I'm trying to run openai_api_demo.py on 2 nvidia 3090 graphics cards. Unfortunately, the current script works great but on 1 card with --quant 4 I would like to be able to use the 2 GPU model with full precision. Thank you in advance for your help

zRzRzRzRzRzRzR commented 3 days ago

check the demo in basic_demo/cli_demo_multi_gpus.py will help