The tokenizer max length should be equal to model max length

weaviate / t2v-transformers-models

This is the repo for the container that holds the models for the text2vec-transformers module

BSD 3-Clause "New" or "Revised" License

39 stars 27 forks source link

The tokenizer max length should be equal to model max length #65

Open Rami-Ismael opened 1 year ago

Rami-Ismael commented 1 year ago

I don't know any model transformer models where the where the model max length is 500. Second, I know that any model with a max length should be equal to power of 2 if it's on a gpu.

https://github.com/weaviate/t2v-transformers-models/blob/7709cdef264d240af71a1c83e8dc1aec540a376e/vectorizer.py#L64C1-L65C64