added option for tokenizer to split on special tokens - Githubissues

allenai / dolma

Data and tools for generating and inspecting OLMo pre-training data.

https://allenai.github.io/dolma/

Apache License 2.0

894 stars 90 forks source link

added option for tokenizer to split on special tokens #176

Closed soldni closed 1 month ago