Open BITEEE0308 opened 1 year ago
Hi Mike,
I finished the code to count the number of tokens for each language using the tokeniser "Tiotoken" from OpenAI and pushed the results and the code to my forked repo. I'm wondering are the results making sense or not in general?
Repo Link: https://github.com/BITEEE0308/modulus-magnus-linguae.git
Hi Mike,
I finished the code to count the number of tokens for each language using the tokeniser "Tiotoken" from OpenAI and pushed the results and the code to my forked repo. I'm wondering are the results making sense or not in general?
Repo Link: https://github.com/BITEEE0308/modulus-magnus-linguae.git