openvinotoolkit / model_server

A scalable inference server for models optimized with OpenVINO™
https://docs.openvino.ai/2024/ovms_what_is_openvino_model_server.html
Apache License 2.0
679 stars 212 forks source link

for rerank export tokenizer with extended max_length to allow documen… #2792

Closed dtrawins closed 3 weeks ago

dtrawins commented 3 weeks ago

…t chunking

🛠 Summary

🧪 Checklist

dtrawins commented 3 weeks ago

closing because max_length will not be added to tokenzer cli in 2024.4