search

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.14k stars 1.88k forks source link

memory usage need to be optimized #163

Open banditsmile opened 2 months ago

banditsmile commented 2 months ago

I use gpt-crawler to crawl a 100k pages project, when crawled pages number goes to 4k, the project cost 4GB memory, meets npm memory limitation.

And the the project can't continue by reload.

It seems that 100k pages will cost 100GB memory.