ai-for-java / ai4j

Apache License 2.0
26 stars 3 forks source link

[Feature] Integrate tokenizer #19

Closed deep-learning-dynamo closed 1 year ago

deep-learning-dynamo commented 1 year ago

As a java developer I want to calculate the number of tokens in a string/document to be able to predict/control the costs.

Check: