Closed silverriver closed 1 year ago
Thank you for your quick response. I have some followup questions regarding to the process used in merging the Chinese and English vocab.
thank you very much in advance
I am wondering if it possible for you to share the script used to merge these vocabs?
Specifically, how to merge the trie in the English and Chinese spm?
I am sorry, they are mostly ipython on-the-fly and I didn't saved them.
Thank you for your replay
Could you provide more details about how these tokenizers are learned?
for example: