is_causal=attention_bias is None

allenai / OLMo

Modeling, training, eval, and inference code for OLMo

https://allenai.org/olmo

Apache License 2.0

4.2k stars 392 forks source link

is_causal=attention_bias is None #598

Open nkkbr opened 1 month ago

nkkbr commented 1 month ago

🐛 Describe the bug

https://github.com/allenai/OLMo/blob/5789cfe32390a0e80417e98285647cb8b41029ae/olmo/model.py#L598-L605

should the line 604 be is_causal=attention_bias is not None ?

Versions

Python 3.10.14

epwalsh commented 1 month ago

Hey @nkkbr, when attention_bias is not None, the attention_bias includes a causal mask, so we don't to set is_causal to True since that would be redundant.