Closed arvinxx closed 1 month ago
The recent update to the project involves adding a new dependency, "gpt-tokenizer," to the package.json
file. This enhancement allows the project to utilize a tokenizer library without modifying any existing functionality. The inclusion of this library expands the project's capabilities, promoting better dependency management while maintaining the integrity of the current codebase.
Files | Change Summary |
---|---|
package.json |
Added new dependency "gpt-tokenizer": { "version": "*" } |
🐇 In the garden of code, a new friend appears,
A tokenizer swift, bringing joy and no fears.
With every new line, our functions will dance,
Embracing the change, let’s give it a chance!
Hops of delight, in our project we play,
For growth and for fun, let’s code night and day! 🌼
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?
gpt-tokenizer : https://github.com/niieani/gpt-tokenizer
this package can count token in browser : https://gpt-tokenizer.dev/
but the cost is the large file size.
Summary by CodeRabbit