search

facebookresearch / MobileLLM

MobileLLM Optimizing Sub-billion Parameter Language Models for On-Device Use Cases. In ICML 2024.
Other
1.16k stars 65 forks source link

Layer sharing model issues #9

Closed pdh930105 closed 4 months ago

pdh930105 commented 4 months ago

Thank you for sharing the code for this research.

I couldn't find the code and config file related to layer sharing (-LS) in the model implementation section.

Will this part be updated in the future?

Atharva-Phatak commented 4 months ago

@pdh930105 If you investigate the code for LlamaForCausalLM/LlamaModel you will be able find both the code for layer sharing and embedding sharing.

pdh930105 commented 4 months ago

@Atharva-Phatak thank you for the info!