In tokenize_single_file.py (line 61), the dtype of data saved in .npy file is set to be uint16. However it is not correct for the case where vocabulary size is large than 65535. It is more safe to set it to uint32, although it doubles the cost of storage.
Thanks for the suggestion! We worked with uint16 to save space for llama2, which only has a vocab of 32000, you are correct that we need to use uint32 for larger vocabs.
In
tokenize_single_file.py
(line 61), the dtype of data saved in .npy file is set to be uint16. However it is not correct for the case where vocabulary size is large than 65535. It is more safe to set it to uint32, although it doubles the cost of storage.