GPU Required for Vision encoder pre-training and Instruction finetuning

opendilab / LMDrive

[CVPR 2024] LMDrive: Closed-Loop End-to-End Driving with Large Language Models

Apache License 2.0

526 stars 48 forks source link

GPU Required for Vision encoder pre-training and Instruction finetuning #49

Open SunHongyang10 opened 2 months ago

SunHongyang10 commented 2 months ago

Hello, doctor Shao, wonderful work!

may I ask if a 24GB 3090 GPU can handle the Vision encoder pre-training and Instruction finetuning?

thank you!

deepcs233 commented 1 month ago

Hi! The Vision encoder pre-training may be proceeded with a 24GB GPU. However, the Instruction fine-tuning stage needs to load a 7B-LLM, which need more GPU memory or more GPU cards.