Cherrysaber
commented
1 month ago # pytorch >= 2.1.0
model = Qwen2AudioForConditionalGeneration.from_pretrained(
"Qwen/Qwen2-Audio-7B", device_map="cuda:0", torch_dtype=torch.bfloat16)❯ nvidia-smi
Fri Sep 6 20:22:58 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 Off | 00000000:01:00.0 Off | Off |
| 30% 47C P2 69W / 450W | 16867MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 Off | 00000000:02:00.0 Off | Off |
| 30% 47C P8 12W / 450W | 22MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
When I was running this, I encountered the problem of gpu out of memory. I am not sure if there is any solution, or if it can actually be run on the 4090 24GB, but I may be doing it wrong.
Thanks.