Closed anthonyduong9 closed 1 month ago
@anthonyduong9 Thanks for adding these keys. Could you fill in the template on your original comment? I know this a draft, but it would be good to have an idea of what is remaining on this to understand what you are all planning.
@anthonyduong9 Thanks for adding these keys. Could you fill in the template on your original comment? I know this a draft, but it would be good to have an idea of what is remaining on this to understand what you are all planning.
@bryce13950 No problem. I've filled out the template. Let me know if anything's unclear.
Beautiful
Description
Adds "n_kv_heads" to the Model Properties Table in the documentation. Some models use multi-query attention (which uses a single key and value head) or grouped-query attention (which uses multiple key and value heads, but less than the number of query heads). This change lets users easily see which models use these.
Fixes # https://github.com/TransformerLensOrg/TransformerLens/issues/522
Type of change
Please delete options that are not relevant.
Screenshots
Please attach before and after screenshots of the change if applicable.
Before
After
Checklist: