Sliding Window Attention

google-deepmind / gemma

Open weights LLM from Google DeepMind.

http://ai.google.dev/gemma

Apache License 2.0

2.37k stars 292 forks source link

Open aniquetahir opened 1 month ago

aniquetahir commented 1 month ago

I noticed this version does not contain sliding window attention needed for Gemma 2 models.

gustheman commented 1 month ago

We are working to release the Gemma 2 related code soon, sorry for the delay

Mddct commented 1 month ago

any update?

canyon289 commented 1 month ago

Hey folks, Thank you for your interest

It will be released soon and contain all the v2 updates such as sliding window attention and GQA. I'll update this issue in the next two weeks!