Mistral: A strong, northwesterly wind: Framework for transparent and accessible large-scale language model training, built with Hugging Face 🤗 Transformers.
Apache License 2.0
562
stars
49
forks
source link
switch to using my pre-detokenized wikitext to make tests a bit faster... #157
the new HF datasets code seems to not be able to compute hash finger prints for detokenization (and when I give it a hashcode, it complains about a lambda inside HF datasets itself!), so we have to detokenize over and over again when we use those libraries.
This PR switches to a pre-detokenized version I made.
dlwh
commented
2 years ago
the new HF datasets code seems to not be able to compute hash finger prints for detokenization (and when I give it a hashcode, it complains about a lambda inside HF datasets itself!), so we have to detokenize over and over again when we use those libraries.
This PR switches to a pre-detokenized version I made.