Are you guys planning to implement GQA?

young-geng / EasyLM

Large language models (LLMs) made easy, EasyLM is a one stop solution for pre-training, finetuning, evaluating and serving LLMs in JAX/Flax.

Apache License 2.0

2.38k stars 254 forks source link

Closed Taekyoon closed 1 year ago

Taekyoon commented 1 year ago

I'm just curious about the development status because I was considering implementing GQA to train with llama2 34, 70b models.

Taekyoon commented 1 year ago

Found this issue related to #77 close this issue.