search

Luodian / Otter

🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
https://otter-ntu.github.io/
MIT License
3.56k stars 241 forks source link

[Feat/Model] add lora training to LLM part. #206

Closed Luodian closed 1 year ago

Luodian commented 1 year ago

Unfreeze LoRA LLM parts and add them into training.

Add 0.008B params into trainable params.

Using LoRA with config:{'lora_alpha': 32, 'lora_dropout': 0.05, 'r': 16}
trainable params: 8388608 || all params: 7773491216 || trainable%: 0.10791300545543706
LoRA trainable param: 0.008 B
Total Trainable param: 1.394 B
Luodian commented 1 year ago

@ZhangYuanhan-AI Please review, especially for instruction_following.py.