caikit / caikit-nlp

Apache License 2.0
12 stars 45 forks source link

Adds tokenization task for embeddings #365

Closed flaviabeo closed 1 month ago

flaviabeo commented 2 months ago

This implements the TokenizeTask for the Embeddings module. Works for all the models of type SentenceTransformer.

cc: @mynhardtburger @markstur