search

nvtransfer / RULER

This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?
Apache License 2.0
646 stars 43 forks source link

lost in the middle problem #24

Closed vkaul11 closed 3 months ago

vkaul11 commented 4 months ago

In the NIAH task do you address the lost in the middle problem ? In the sense that can we control that the needles are inserted only in the middle and not in the beginning and the end because that seems to be where the hardness of the problem lies?

hsiehjackson commented 4 months ago

In our NIAH, we insert needles in random positions. If you want to test lost in the middle, you can change positions here. In our experiments, we found the middle issue got more serious as sequence length increases. If we put needle at the beginning or at the end, we may only see slight degradations. Hence, the degradation from most of long-context LLMs are because they tend to lost in the middle as sequence length increases.

muhangao commented 3 weeks ago

Do you still observe Lost-in-the-Middle in Llama3.1 8B? I'm using RULER and it seems like the performance only degrades when the needle is at beginning.