search

princeton-nlp / LLM-Shearing

[ICLR 2024] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
https://arxiv.org/abs/2310.06694
MIT License
533 stars 39 forks source link

Release sheared model without re-training? #44

Closed sbwww closed 7 months ago

sbwww commented 8 months ago

Hello, will you release sheared models without re-training? Currently, there are only some graphs showing the averaged ACC of each training checkpoint. I think it would be instructive if we can evaulate the un-trained model!

xiamengzhou commented 8 months ago

Do you mean the pruned models before training? Would love to release those if they are helpful. Stay tuned, I will get them uploaded within the week!

sbwww commented 7 months ago

Is there any update? Thanks in advance. @xiamengzhou

xiamengzhou commented 7 months ago

Apologies for the delay! You can find the two pruned models on HF now. princeton-nlp/Sheared-LLaMA-1.3B-Pruned, princeton-nlp/Sheared-LLaMA-2.7B-Pruned

sbwww commented 7 months ago

Thank you very much!