search

Yaxin9Luo / Gamma-MOD

Officail Repo of γ -MOD: Mixture-of-Depth Adaptation for Multimodal Large Language Models
https://yaxin9luo.github.io/gamma-mod-webpage/
Other
21 stars 2 forks source link

I would like to know which files' code you modified? #2

Open FanshuoZeng opened 1 month ago

FanshuoZeng commented 1 month ago

This is a very meaningful work. I would like to know which files' code you modified? Could you please tell me?

Yaxin9Luo commented 1 month ago

Thanks for your attention, gamma_mod/model/language_model/llava_llama_mod.py , gamma_mod/model/builder.py and gamma_mod/train/train.py are the three most important files. In general, you can adapt our method to any MLLMs and you only need to modify these three main files. But be aware that some MLLMs have different transformer versions, different backbone LLMs. For example, in gamma_mod/MGM/mgm/model/language_model/mod_mgm_llama.py, I also add codes to handle the position embedding and cache modifications caused by the transformers version ==4.36.2. In addition, if you wish to change llama to other LLMs, you also need to deal with their conversation template in gamma_mod/conversation.py