search

eole-nlp / eole

Open language modeling toolkit based on PyTorch
https://eole-nlp.github.io/eole
MIT License
24 stars 6 forks source link

Fix the tokenizer saving in the HF converter #14

Closed l-k-11235 closed 1 month ago

l-k-11235 commented 1 month ago

With this small fix, eole convert HF will save the tokenizer to the --output directory, when it is downloaded from a huggingface folder.