Add gpt-tokenizer package and implement size and token limits

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

https://www.builder.io/blog/custom-gpt

ISC License

18.59k stars 1.97k forks source link

Add gpt-tokenizer package and implement size and token limits #62

Closed guillermoscript closed 10 months ago

guillermoscript commented 10 months ago

This pull request adds the gpt-tokenizer package to the project and implements optional size and token limits in the Config interface. The maxFileSize property allows specifying the maximum file size in bytes to include in the output file, and the maxTokens property allows specifying the maximum number of tokens to include in the output file.

Fixes #44 #77

github-actions[bot] commented 9 months ago

:tada: This PR is included in version 1.2.0 :tada:

The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket: