search

ibm-granite / granite-code-models

Granite Code Models: A Family of Open Foundation Models for Code Intelligence
https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330
Apache License 2.0
1.06k stars 73 forks source link

GQA? #4

Closed ramzeez88 closed 4 months ago

ramzeez88 commented 4 months ago

Have the granite models been trained with grouped query attention?

mayank31398 commented 4 months ago

Hi, the 8b model uses GQA more info here:

Screenshot 2024-05-08 at 2 13 30 PM
mayank31398 commented 4 months ago

closing this