issues
search
dragosrotaru
/
ppeforfree
Collective sensemaking for mutual aid groups manufacturing PPE during COVID.
https://ppeforfree.org
GNU General Public License v3.0
5
stars
5
forks
source link
Make the Scraper Reliable
#39
Open
dragosrotaru
opened
4 years ago
dragosrotaru
commented
4 years ago
[ ] Implement partial scraping - recent members only, detect field change frequency, prioritize based on group size, history of growth
[ ] don't do batch jobs, let the scraper run as a chron job or background task
[ ] use counts vs list.length to detect issues
[ ] save stderr and scraper parameters
[x] connect to existing browser or save session (no relogin)
[x] make auto-scrolling work by waiting for network idle
[x] create phony public and private groups for integration tests
[ ] deploy scraper on multiple personal computers (Scraping@Home V0.0.1)
[ ] use exponential backoffs