Closed leviethung2103 closed 10 months ago
Hello,
The model has been trained on the A100 GPU. However, I am wondering about the GPU memory cost during inference.
Currently, I have a 3060 GPU with 12 VRAM. Can it be used for running inference?
Thank you
You can just load the model in 8-bit, taking about 7.5GB of memory: load_in_8bit=True https://huggingface.co/docs/transformers/main/en/main_classes/quantization
load_in_8bit=True
Hello,
The model has been trained on the A100 GPU. However, I am wondering about the GPU memory cost during inference.
Currently, I have a 3060 GPU with 12 VRAM. Can it be used for running inference?
Thank you