Closed jettjaniak closed 2 months ago
https://huggingface.co/delphi-suite/stories-tokenizer is a result of scripts/train_tokenizer.py -i delphi-suite/stories -f story -s "train" -o delphi-suite/stories-tokenizer -v 4096 -t hf_... and https://huggingface.co/datasets/delphi-suite/stories-tokenized was tokenized using it (see #106 for details)
scripts/train_tokenizer.py -i delphi-suite/stories -f story -s "train" -o delphi-suite/stories-tokenizer -v 4096 -t hf_...
https://huggingface.co/delphi-suite/stories-tokenizer is a result of
scripts/train_tokenizer.py -i delphi-suite/stories -f story -s "train" -o delphi-suite/stories-tokenizer -v 4096 -t hf_...and https://huggingface.co/datasets/delphi-suite/stories-tokenized was tokenized using it (see #106 for details)