Closed Arbor334 closed 6 months ago
Hello, @Arbor334. It's correct to tokenize the long sequence and split it into short chunks.
thanks @18907305772 ,If the above splitting of the data set is completed, I will implement the different scripts below, corresponding to
llama
, open_llama
mpt
, respectively.
That's right. It is recommended to test the code on a small subset before running it on the entire dataset.
Thank you @18907305772 , I will adopt your suggestion
When I change the path in split_long_text.py to my own directory, i got![2024-03-09-09-50-06-image](https://github.com/18907305772/FuseLLM/assets/57646922/a77cf644-c0d0-4745-949a-97f6e8365080)
is it right?