Update redwood / Add Docker / Fix Scraper

lachlanjc / predictcovid

Visualize & track the 2020 COVID-19 pandemic by country.

https://predictcovid.netlify.app/

MIT License

112 stars 11 forks source link

Update redwood / Add Docker / Fix Scraper #13

Open bozdoz opened 3 years ago

bozdoz commented 3 years ago

I had issues bringing up the project and I realized everything was out-of-date. Ran yarn rw update, and compared with a brand new project to fix up any issues.

Also began fixing the scrape function. Appears to be getting the same data for each graph. Maybe a WIP:

netlify[bot] commented 3 years ago

Deploy request for countrycovid19 pending review.

Review with commit 36c24b12fe383d1b1d6f07932c938afc52f46990

https://app.netlify.com/sites/countrycovid19/deploys

vercel[bot] commented 3 years ago

Someone is attempting to deploy a commit to a Personal Account owned by @lachlanjc on Vercel.

@lachlanjc first needs to authorize it.

bozdoz commented 3 years ago

Resolves #12

vercel[bot] commented 3 years ago

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/lachlanjc/covid19/ASmkjC8tyoKFtPo4x7nKrNiDcuwJ
✅ Preview: https://covid19-git-fork-bozdoz-update-redwood-lachlanjc.vercel.app

lachlanjc commented 3 years ago

Wow, thanks for bringing this back to life!! Yeah, this was one of the first releases of Redwood we ran it on. Some observations/questions:

Looks like netlify.toml needs an update — the build logs say the db up command has changed to prisma migrate
We implemented the redwood-hack because the database transactions wouldn't run without it, can we drop that now?
Since this project is just using JS without TS, & no authentication, why are the new types files needed?
How does the new scraping work? We just go to the URL of that function to update the database? Will it backfill to when I stopped manually updating it last summer?
Looks like the chart data fetching isn't working in prod: https://covid19-qhsp5dim1-lachlanjc.vercel.app/

bozdoz commented 3 years ago

Hmm, that's curious.

I will try the site out in docker, to make sure there's a straight-forward approach to getting this set up. That should help figure out netlify (don't have much experience with it).

I imagine redwood-hack is useless, but I'll test it out in docker.

Those new types files were added automatically (and updated frequently!). Might be worth adding them to .gitignore.

New scraping just altered the selector for the script tag, and parsed the data differently; that's it. shouldn't be anything new. Maybe the script tags are in a different format now, and maybe that's why the daily stats numbers are in error. I'm not sure what it was doing before though.

lachlanjc commented 3 years ago

Interesting. Yeah, let's ignore the types. The scraper never worked consistently—that was a project we started with the hopes the site could update itself, but I ended up just manually entering the numbers into the database using SQL every day until last summer, when the visualization wasn't that useful anymore.

bozdoz commented 3 years ago

Docker was a bit tough, but I got it up eventually.

Added a few more countries. Changed default country to be an environment variable. Caching one of the queries to prevent re-renders. Removed redwood-hack directory and seems to work fine. Also I think I fixed the scraper.

Try it out if you have docker: docker-compose up --build

Scraper seems to take about 10min

bozdoz commented 3 years ago

proof of scraper fix (I think): 🤔