search

johndpjr / AgTern

19 stars 5 forks source link

Fix web scraper memory leak #87

Closed johndpjr closed 1 year ago

johndpjr commented 1 year ago

Closes #68

Replaces the close() on the Chrome webdriver with quit(). close exits the focused window, while quit will actually free the resources. close might do this already if there's one window open, but it's better to actually quit and explicitly free all resources. I looked at the system monitor and quit is freeing all chrome instances as expected (close is doing the same, but there might be circumstances that I was missing). I did my tests on Linux

JeremyEastham commented 1 year ago

Did you test that the resources are still deallocated if the program crashes? You could test this by putting raise Exception() somewhere in the scraping code (maybe WebScraper.scrape_company).

johndpjr commented 1 year ago

Did you test that the resources are still deallocated if the program crashes? You could test this by putting raise Exception() somewhere in the scraping code (maybe WebScraper.scrape_company).

I tried raising exceptions in both close and quit and saw that it deallocated all resources in both cases. Maybe you had other chrome tabs open when you saw this issue...