cohere-ai / cohere-typescript

The Cohere TypeScript SDK
https://docs.cohere.ai
MIT License
122 stars 18 forks source link

Add "token_strings" to the response of Tokenize #43

Closed mkozakov closed 2 years ago

mkozakov commented 2 years ago

The API response of tokenize now includes a token_strings field. This PR adds it to the SDK.

Example:


const response = await cohere.tokenize({
  text: 'hello world'
});

console.log(response.body["tokens"]) // [33555, 1114]
console.log(response.body["token_strings"]) // ["hello", " world"]