Is 'tiktokn' in requirements.txt meant to be 'tiktoken'?

baaivision / Emu3

Next-Token Prediction is All You Need

Apache License 2.0

1.81k stars 71 forks source link

Is 'tiktokn' in requirements.txt meant to be 'tiktoken'? #4

Open WingLoong233 opened 1 month ago

WingLoong233 commented 1 month ago

I noticed an issue while trying to install the project dependencies and wanted to check if there might be a small typo in the requirements.txt file.

Is it possible that 'tiktokn' in the requirements.txt file should actually be 'tiktoken'? The latter is a tokenizer package developed by OpenAI for use with GPT models.

ryanzhangfan commented 1 month ago

Sorry for the typo. It's indeed "tiktoken" pkg. We've fix it and updated requriements.txt