AnswerDotAI / bert24

Apache License 2.0
25 stars 3 forks source link

Allow resizing the model embeddings layer to fit the tokenizer #48

Closed NohTow closed 1 month ago

NohTow commented 1 month ago

Since the tokenizer has not the same number of tokens than the base model embeddings, we need to allow resizing it.

I insist, but maybe it's better we get ride of anything that is set as default by the base model and explicitly define everything in the config.