create worker to scrape - Githubissues

bozdoz / squint

Criticize your financial losses

MIT License

1 stars 0 forks source link

create worker to scrape #3

Open bozdoz opened 4 months ago

bozdoz commented 4 months ago

Two variants: (1) Live scraper (web sockets) and (2) cron job scraper (redis & db?)

Live scraper

initial auth is done here
UI needs to prompt the worker (always redis?)
UI needs to join a websocket (public?)?

Cron Scraper

subsequent auth is done here, maybe weekly, or every two days
needs to notify/email when there's a failure and never try again until it's fixed
worker needs to listen to redis queue
start browser, and scrape data