winglian
commented
2 months ago The default is eager attention when leaving both flash_attention and sdp_attention unset in the yaml.
Closed dameikle closed 2 months ago
winglian
commented
2 months ago The default is eager attention when leaving both flash_attention and sdp_attention unset in the yaml.
Description
Adds eager_attention to AxolotlInputConfig so it can be set in the config files
Motivation and Context
When fine-tuning Gemma 2 there is a warning on the attention mechanism used, with eager attention strongly recommended. This allows it to be explicitly set, allowing users to pick between using the flash attention or eager attention.