Things you can do with the token embeddings of an LLM
MIT License
1k
stars
29
forks
source link
tokenizer = Tokenizer.from_file(str(tokenizer_path)) Exception: data did not match any variant of untagged enum PyNormalizerTypeWrapper at line 49 column 3 #16
tokenizer = Tokenizer.from_file(str(tokenizer_path))
Exception: data did not match any variant of untagged enum PyNormalizerTypeWrapper at line 49 column 3
gfkdliucheng
commented
2 hours ago
tokenizer = Tokenizer.from_file(str(tokenizer_path)) Exception: data did not match any variant of untagged enum PyNormalizerTypeWrapper at line 49 column 3