search

g0v / vaccinate

vaxx.tw is a tool to find COVID-19 Vaccine Appointments near you
http://vaxx.tw
MIT License
33 stars 20 forks source link

Docker for scraper #71

Closed yorkxin closed 3 years ago

yorkxin commented 3 years ago

This patch tries to make scraper a standalone Docker image.

Specifically:

  1. local_scraper.py can now output to JSON using --out=file.json arg.
  2. local_scraper.py can still store results to Redis, but needs --redis arg (breaking change!!)
    • env vars remain same, but I added REDIS_SSL flag env var for local testing. Default to yes; set to no to disable SSL.

My next plan:

  1. Run this Docker image on AWS ECS as a periodic task, then send the result to S3.
  2. Developer can run this script without setting up a Redis instance locally. Faster iteration.
  3. Eventually, frontend can download live data from S3 (maybe add a CDN in front), to eliminate backend completely.

Note that this PR replaces my previous PR #55, which is too aggressive to be useful.

chrisbobbe commented 3 years ago

It looks like GitHub Actions found some failing checks; what's going on with those? 🙂

yorkxin commented 3 years ago

hmm apparently I didn't run type checker locally. I'll fix them. However it seems that my patch is out-of-sync from the current master branch. I'll need some time to fix them.

yorkxin commented 3 years ago

I have to close this now because the system architecture seems to have changed a lot. I have to do this from scratch again.