PlaywrightCrawler memory problem and errors

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

https://www.builder.io/blog/custom-gpt

ISC License

18.14k stars 1.88k forks source link

PlaywrightCrawler memory problem and errors #162

Open lipstk opened 2 months ago

lipstk commented 2 months ago

Hi, crawling some website PlaywrightCrawler shows some errors and probably goes out of memory: log.txt

There is a: WARN PlaywrightCrawler:AutoscaledPoolSnapshotter: Memory is critically overloaded. Using 2014 MB of 983 MB (205%). Consider increasing available memory.

How to increase memory? Where 983MB of memory are used? The VM has 4GB of memory. Thanks in advance,