search

TencentARC / LLaMA-Pro

[ACL 2024] Progressive LLaMA with Block Expansion.
https://tencentarc.github.io/LLaMA-Pro/
Apache License 2.0
482 stars 35 forks source link

Code for training llama pro? #2

Open yhyu13 opened 10 months ago

yhyu13 commented 10 months ago

Hi,

If I get it correctly, you have used code from https://github.com/allenai/open-instruct as base.

Would you release the full code of reproducing llama2 pro 8B?

Thanks!

hills-code commented 10 months ago

Yes, of course. I will organize the code recently. Thanks for your interest.

raghavgarg97 commented 10 months ago

Hey @hills-code ,could you also add code for converting a model by adding identity blocks for training ? I am excited to use similar techniques for other open-source models like Qwen!! Thanks!:)

hills-code commented 10 months ago

Hey @hills-code ,could you also add code for converting a model by adding identity blocks for training ? I am excited to use similar techniques for other open-source models like Qwen!! Thanks!:)

I have added the block expansion script under the folder scripts. You can check it for reference. Hope it will be helpful!

raghavgarg97 commented 10 months ago

i will check that out,Thanks!

raghavgarg97 commented 10 months ago

@hills-code i was able to get it to work! Btw you only train the added layers and not even the lm head? and also do you think directly going for SFT by skipping pretraining will work?

yhyu13 commented 10 months ago

@raghavgarg97 No, I don't think you can skip the "post-training" for extra blocks on some new corpus (e.g. bigcode, as mentioned in the paper) before applying tuning bc the purpose of LLaMA Pro is sort of "add new ability w/o forgetting by introducing more layers". The new ability gained is training new layers on new corpus.

Abolfazl-kr commented 10 months ago

does anybody know how to continue pre-training LLaMA Pro? @yhyu13 @raghavgarg97 @yxgeee @hills-code

yhyu13 commented 10 months ago

nah，code has not been released yet