Open rasbt opened 1 month ago
One note: tests for Mistral model (regular model, adapter and LoRA) should also be updated to test functionality of a sliding window attention, meaning that the size of an input should be larger than the size of a sliding window. Otherwise, we will gate false positives. It can be adapted from the corresponding tests for Gemma 2.
Opening this issue so we don't forget: Once #1545 is merged, let's also add sliding window attention to Mistral 0.1