How to save tokenizer automatically?

[arlethitgo]

Hello! I would like to ask how do you automatically save the tokenizer of the new finetuned models?

[giove91]

Hi, thanks for your interest in our work! We do not change the tokenizer, so it is exactly the same as the pre-trained tokenizer. The delimiters of the "augmented natural languages" are not treated in any special way during tokenization.

I hope this clarifies!