search

Alpha-VLLM / Lumina-mGPT

Official Implementation of "Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining"
https://arxiv.org/abs/2408.02657
507 stars 22 forks source link

training with lora？ #30

Closed xiexing0916 closed 1 month ago

xiexing0916 commented 1 month ago

Is there a way (lora?) to train with less memory burden? Full fine-tuning is not trainable at 48G of RAM.

ChrisLiu6 commented 1 month ago

The released codes do not support lora yet, but it is easy to implement and you can do it by yourself. You may also implement the get_trainable_params method for the model class to only train part of the model parameters.

https://github.com/Alpha-VLLM/Lumina-mGPT/blob/104abe453ec1acca5863698629c4db2111b0b3fc/xllmx/solvers/finetune/finetune.py#L273

The follows issues may be helpful:

https://github.com/Alpha-VLLM/Lumina-mGPT/issues/17 https://github.com/Alpha-VLLM/Lumina-mGPT/issues/18

xiexing0916 commented 1 month ago

The released codes do not support lora yet, but it is easy to implement and you can do it by yourself. You may also implement the get_trainable_params method for the model class to only train part of the model parameters.

https://github.com/Alpha-VLLM/Lumina-mGPT/blob/104abe453ec1acca5863698629c4db2111b0b3fc/xllmx/solvers/finetune/finetune.py#L273

The follows issues may be helpful:

17 #18

Thanks for the answer! I would also like to ask how much memory is needed to fine tune all the parameters? I currently can't work with 3 48G A6000.

ChrisLiu6 commented 1 month ago

Thanks for the answer! I would also like to ask how much memory is needed to fine tune all the parameters? I currently can't work with 3 48G A6000.

We have not tested on the memory lower bound. 8A100 80G should be enough, but the lower I have no idea😅.