Implement eq in SentencePieceModel based on getstate
This allows comparing two vocabularies without loading the model. Currently, eq is implemented by comparing the md5 checksum of the loaded models, which requires the model to be loaded. It also ignores other parameters of the vocabulary such as extra_ids, reverse_extra_ids, etc.
Implement eq in SentencePieceModel based on getstate
This allows comparing two vocabularies without loading the model. Currently, eq is implemented by comparing the md5 checksum of the loaded models, which requires the model to be loaded. It also ignores other parameters of the vocabulary such as
extra_ids,reverse_extra_ids, etc.