search

scrapy-plugins / scrapy-playwright

🎭 Playwright integration for Scrapy
BSD 3-Clause "New" or "Revised" License
1.03k stars 113 forks source link

RAM usage question #209

Closed andrei-volkau closed 1 year ago

andrei-volkau commented 1 year ago

I have one question about RAM usage.

Screenshot 2023-07-16 at 15 36 37

My scrapy code is not executing anymore, but I am still able to see that RAM is being consumed by playwright. What should I do to avoid all RAM to be consumed by paywirght even so that scraping process is not running any more? Should I find and kill playwright processes on Spider closed event?

I was having a lot of playwright timeouts during scraping. Maybe I experienced memory leaks?

elacuesta commented 1 year ago

A good catch, thank you for noticing. I suppose #214 should be enough.