Add 'auto truncating' feature.

ksanman / AllMiniLML6v2Sharp

net standard 2.1 Library for running Sentence Transformers All-MiniLM-L6-v2 from C#.

MIT License

12 stars 6 forks source link

Closed ksanman closed 5 months ago

ksanman commented 5 months ago

If truncate is true, truncate to 512 tokens.
Add tests
Fix issue with bert tokenizer where a maxSequence smaller than the tokenized sentence breaks encoding. fixes #3