mikeizbicki / modulus-magnus-linguae

8 stars 6 forks source link

OpenAI Count Accurate or Not. #5

Open BITEEE0308 opened 1 year ago

BITEEE0308 commented 1 year ago

Hi Mike,

I finished the code to count the number of tokens for each language using the tokeniser "Tiotoken" from OpenAI and pushed the results and the code to my forked repo. I'm wondering are the results making sense or not in general?

Repo Link: https://github.com/BITEEE0308/modulus-magnus-linguae.git