search

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.14k stars 1.88k forks source link

How to paginate for large JSON files? #119

Open chnsh opened 6 months ago

chnsh commented 6 months ago

I get an error that the JSON files are too large for GPT. I can use maxTokens, maxFileSize to reduce it but how do I create the next JSON file such that it starts off from the current one?