Feature request: Display token count when generating text files

abinthomasonline / repo2txt

Web-based tool converts GitHub repository contents into a single formatted text file

https://repo2txt.simplebasedomain.com/

MIT License

875 stars 87 forks source link

Feature request: Display token count when generating text files #12

Open doggy8088 opened 2 weeks ago

doggy8088 commented 2 weeks ago

Is it possible to display token count when generating text files?

abinthomasonline commented 2 weeks ago

Great suggestion—this would be a helpful addition!

There are two ways we could implement it: a rough estimate using the rule of 4 characters ≈ 1 token, or by using a library like gpt-tokenizer for more accurate counting.

Do you think it's worth adding this library and loading its vocabulary for the extra precision?