Hi! Would tiktoken be a good library to utilize when counting gpt tokens? I tried to run it using this sentence from the Bible in Achuar. Tiktoken counts the sentence as being 41 tokens (not sure if that is a reasonable count for gpt-3.5)
"Tura Judá Tamaran nuwatak, ni uchirin Faresan tura chikich uchirin Zara naartinun yajutmarmiayi." Thank you!
Hi! Would tiktoken be a good library to utilize when counting gpt tokens? I tried to run it using this sentence from the Bible in Achuar. Tiktoken counts the sentence as being 41 tokens (not sure if that is a reasonable count for gpt-3.5) "Tura Judá Tamaran nuwatak, ni uchirin Faresan tura chikich uchirin Zara naartinun yajutmarmiayi." Thank you!