search

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.15k stars 1.88k forks source link

Size #77

Open albrox opened 7 months ago

albrox commented 7 months ago

I have successfully crawled a whole website and have the output file as JSON.

The problem is that the file size is 93MB and after uploading it to ChatGPT I get an error message stating that the file is too large. Is there a known limit in size that can be uploaded and can this be chunked into different parts?

guillermoscript commented 7 months ago

hello @albrox please look at my PR it just about that same problem

learner1999 commented 7 months ago

@guillermoscript Thank you very much for your PR. I would like to ask you what is the appropriate size for a single file. I have not found the official instructions of openai.

guillermoscript commented 7 months ago

hello @learner1999 sorry but I also don't know a recommended size per file, that's going to be trial an error for you use case and with the PR you can change the size to fit your needs