add serialization for `ignore_merges`

huggingface / tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

https://huggingface.co/docs/tokenizers

Apache License 2.0

8.68k stars 745 forks source link

add serialization for `ignore_merges` #1504

Closed ArthurZucker closed 2 months ago

ArthurZucker commented 2 months ago

Make sure we serialize ignore_merges

HuggingFaceDocBuilderDev commented 2 months ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.