search

caikit / caikit-nlp

Apache License 2.0
12 stars 45 forks source link

duplicate PR for tokenization tasks request #331

Closed swith004 closed 6 months ago

swith004 commented 6 months ago

duplicate PR of 325 , which Added tokenization tasks in order to serve tokenization request and return the token count