Closed vkaul11 closed 3 months ago
In our NIAH, we insert needles in random positions. If you want to test lost in the middle, you can change positions here. In our experiments, we found the middle issue got more serious as sequence length increases. If we put needle at the beginning or at the end, we may only see slight degradations. Hence, the degradation from most of long-context LLMs are because they tend to lost in the middle as sequence length increases.
Do you still observe Lost-in-the-Middle in Llama3.1 8B? I'm using RULER and it seems like the performance only degrades when the needle is at beginning.
In the NIAH task do you address the lost in the middle problem ? In the sense that can we control that the needles are inserted only in the middle and not in the beginning and the end because that seems to be where the hardness of the problem lies?