App data retriever memory leak - Blows up scraping scraping app data.

sociam / xray-archiver

Mobile app archiver built for the SOCIAM xray project. This repo contains functionality to generate search terms, scrape app data and download APK's from the google play store.

BSD 3-Clause "New" or "Revised" License

6 stars 5 forks source link

App data retriever memory leak - Blows up scraping scraping app data. #94

Open AdamSlack opened 6 years ago

AdamSlack commented 6 years ago

Retriever fetches over 500k search terms and then collects 120 sets of app data for each search term. JS then ran out of memory. Rip.

AdamSlack commented 6 years ago

fixed, doing it a search term at a time now. donezo.

AdamSlack commented 6 years ago

Promises not releasing memory. What is life.

sauyon commented 6 years ago

incredible

AdamSlack commented 6 years ago

potentially narrowed down to two things.

One: However i'm going over all search terms in the DB is leading to there being one massive long promise chain that mean the GC can't do its job. I've tried doing it a couple of ways now. each of them lead to mem leak.

Two: The promises aren't actually the issue and its the db.js causing the issues. the pooling for postgres client connections might not be releasing properly when complete. I think its more likely to be One though.