search

NVIDIA / RULER

This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?
Apache License 2.0
738 stars 47 forks source link

Is there any issue in extending context length to 1 million using your script #26

Closed vkaul11 closed 4 months ago

vkaul11 commented 5 months ago

Just checking if there is any reason to restrict the use to 128k or can we use the script to 1 million tokens also ? Secondly is there a NOTICE file we have to use if we modify the code and use it somewhere?

hsiehjackson commented 5 months ago

You can definitely use our script to test 1 million tokens! A potential problem is the speed to generate dataset files can be slow. I think we can use a better solution (binary search) to find the size of distraction since we only linearly increasing it for now in here.

For the NOTICE file, you can copy Apache-2.0 LICENSE.