This PR adds a config option to enable Flash Attention 2's deterministic mode. Determinism will slow down FA2, so I have disabled it by default for pretraining configs but enabled it for all evaluation configs.
Tests
[x] Is the new feature tested? (Not always necessary for all changes -- just adding to the checklist to keep track)
Changes
This PR adds a config option to enable Flash Attention 2's deterministic mode. Determinism will slow down FA2, so I have disabled it by default for pretraining configs but enabled it for all evaluation configs.
Tests