Can you offer the `sentencepiece_model.proto`?

THUDM / icetk

A unified tokenization tool for Images, Chinese and English.

150 stars 17 forks source link

Can you offer the `sentencepiece_model.proto`? #13

Open workingloong opened 6 months ago

workingloong commented 6 months ago

Now, the sentencepiece_model_pb2.py is generated by the low version protobuf which is incompatible with many libs with high version protobuf. We can re-generate the sentencepiece_model_pb2.py with the high version of protobuf if we have the entencepiece_model.proto.