search

mlpc-ucsd / BLIVA

(AAAI 2024) BLIVA: A Simple Multimodal LLM for Better Handling of Text-rich Visual Questions
https://arxiv.org/abs/2308.09936
BSD 3-Clause "New" or "Revised" License
269 stars 28 forks source link

Cuda Out of Memory! #13

Open Ethantequila opened 1 year ago

Ethantequila commented 1 year ago

Hi I try to run a demo(BLIVA_Vicuna 7B) in my local machine(V100, 16GB), It comes with a OOM error: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 86.00 MiB (GPU 0; 15.77 GiB total capacity; 12.18 GiB already allocated; 54.88 MiB free; 12.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

So how much is the minimum amount of GPU memory required ? Or is there any way to reduce GPU usage?

Thanks a lot!

gordonhu608 commented 1 year ago

Thank you for your interest in our work. Short answer is it needs 20G memory. Longer answer is here: https://github.com/mlpc-ucsd/BLIVA/issues/3

Ethantequila commented 1 year ago

Thank you for your interest in our work. Short answer is it needs 20G memory. Longer answer is here: #3

Thanks for your quick reply! I am wondering how much GPU memory needed for run BLIVA_Vicuna 7B with INT8?