Support for Llama-3 / GQA?

princeton-nlp / LLM-Shearing

[ICLR 2024] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

https://arxiv.org/abs/2310.06694

MIT License

533 stars 39 forks source link

Closed LorrinWWW closed 3 months ago

LorrinWWW commented 4 months ago

Would it be possible to support llama-3? I think the missing part is GQA. Thank you so much! Related: #62

xiamengzhou commented 3 months ago

We're working on it. Thanks for your inquiry!