Open KirilAngelov opened 1 month ago
我测试了最新的代码,int4模型一共只占用了10G显存。但是我估计是在载入的时候超过16G,你尝试修改成内存载入后在移动到显存,而不是直接在显存载入
Hello, Thank you for the information. I am not sure how to load the model and then transfer it to video memory. I will appreciate any example or help in order to run the model. Also, it would be nice to include these instructions for moving the model to video memory after loading it in the official documentation, so users can successfully run the example scripts.
System Info / 系統信息
Hello, I have problems running CogVLM2 using the provided T4 instance in Google Colab. Here are the python modules versions: cuda 12.2 transformers 4.41.2 xformers 0.0.26.post1
Uploading a single JPG image, setting the right image path and prompt results in CUDA out of memory.
I have tried running both the THUDM/cogvlm2-llama3-chat-19B and THUDM/cogvlm2-llama3-chat-19B-int4 using 4bit quantization for both and always have this error: OutOfMemoryError: CUDA out of memory. Tried to allocate 1.11 GiB. GPU
Here is some additional information:
![image](https://github.com/THUDM/CogVLM2/assets/44143480/e9f5285f-e72e-4ba6-ac28-5feb2800d0e4)
Who can help? / 谁可以帮助到您?
No response
Information / 问题信息
Reproduction / 复现过程
Expected behavior / 期待表现
The expected behavior should be CogVLM2 running, because in the official documentation is said that when using 4bit quantization, the model should run on a GPU with 16GB memory.