Leaking memory - Githubissues

gosom / google-maps-scraper

scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

MIT License

879 stars 120 forks source link

Leaking memory #82

Closed sbOogway closed 1 hour ago

sbOogway commented 2 hours ago

Hi, Great project bro very well done but in my setup it leaks memory. each time i run it creates a new chrome process . a simple sudo pkill chrome gave some memory back.

chrome_ps

gosom commented 1 hour ago

@sbOogway can you please give me some more information?

OS
how you start the program ?

sbOogway commented 1 hour ago

i am in debian and running it through docker. every time i run it, it spawns some (from 6 to 18) threads of chrome in the root directory. should nt be they closed when the scraping job is finished? chrome-ps2

this is the form data i am passing. i am calling the api from python. even when i run through the browser it spawns threads like in python payload = "name=test&keywords=bar%20in%20calabria&lang=en&zoom=0&latitude=0&longitude=0&depth=3&maxtime=10m"

now they re more

gosom commented 1 hour ago

how often you call it? it requires around 3m to cleanup.

I assume that you have a web server or similar and you make a request. Do you wait for the program to exist or you sent multiple requests?

when you start the program as it is intended to get started is the behavior the same?

sbOogway commented 1 hour ago

@gosom

i restarted the docker container and i run it through the web interface and i get the same problem. i have waited for each job to finish.

gosom commented 1 hour ago

can you confirm the following:

(1) start the app using the provided docker container in the latest version (2) the web interface is accessible at localhost:8080 (3) you add a keyword in the form and click the start button (4) you wait until the job finishes (5) you click download CSV and it's not empty (6) after these you can still see open chromium browsers?

sbOogway commented 1 hour ago

yes. i dont click the download button. i only start the job. why the chromium browsers run in root directory?

gosom commented 1 hour ago

this is in the docker container not in your host. docker run as root.

let me check if the same happens to me (Fedora) and let you know

sbOogway commented 1 hour ago

ok thanks bro, appreciate it.

gosom commented 1 hour ago

docker pull gosom/google-maps-scraper
mkdir -p gmapsdata && docker run -v $PWD/gmapsdata:/gmapsdata -p 8080:8080 gosom/google-maps-scraper -web -data-folder /gmapsdata

I run the above.

the browsers cleanup up

sbOogway commented 1 hour ago

ok thanks for the time. i still get the problem but i can work it around with pkill. thanks a lot