add sanitation of input text

CdC-SI / eak-copilot

The official repository of the EAK-Copilot project as part of the Innovation Fellowship 2024.

https://cdc-si.github.io/eak-copilot/

GNU General Public License v3.0

3 stars 0 forks source link

Open K-Schubert opened 2 weeks ago

K-Schubert commented 2 weeks ago

Description

Depending on embedding model max token input, truncate input text to model max tokens.