covidatlas / coronadatascraper

COVID-19 Coronavirus data scraped from government and curated data sources.
https://coronadatascraper.com
BSD 2-Clause "Simplified" License
363 stars 179 forks source link

Change CDS build to pull down reports from Li v1 folder #1064

Closed jzohrab closed 4 years ago

jzohrab commented 4 years ago

Currently, CDS runs its own set of scrapers to get and scrape data. We are moving over to Li and eventually will likely convert the existing CDS coronadatascraper.com site to serverless; however, until then, we should stop generating reports in CDS, and just pull them down from Li.

jzohrab commented 4 years ago

This has been merged to master, now waiting for it to be launched to prod.

jzohrab commented 4 years ago

Done and looks good.

TomGoBravo commented 4 years ago

@jzohrab Does this mean that https://coronadatascraper.com/timeseries.csv.zip is now a product of project Li? I'm trying to make sure CAN has switched to Li without breaking anything on our side. Is there a stable URL for fetching the project Li v1 timeseries data as CSV?

jzohrab commented 4 years ago

Hi @TomGoBravo - yes, those reports are now generated from Li. That report is currently at https://liproduction-reportsbucket-bhk8fnhv1s76.s3-us-west-1.amazonaws.com/v1/latest/timeseries.csv

The link is at https://covidatlas.com/data along with the other reports.

The timeseries.csv should be pretty much the same ... see the docs for that at https://github.com/covidatlas/li/blob/master/docs/reports-migrating-from-cds-to-li.md#timeseriescsv. The fields are all there (except for growth factor ... an oversight!), and some other fields added.

TomGoBravo commented 4 years ago

Cool, thank you and congrats on getting this out.