search

covidatlas / li

Next-generation serverless crawler for COVID-19 data
Apache License 2.0
57 stars 33 forks source link

Validate reports still work. #368

Open jzohrab opened 4 years ago

jzohrab commented 4 years ago

With #356 and #365, we'll have a lot more US counties getting included in the baseData.json report. Ensure everything still generates ok, and doesn't time out. (Potentially rewrite this to handle the increased count.)

jzohrab commented 4 years ago

Have merged in https://github.com/covidatlas/li/pull/369 "parallelize base data", which also does recursive get for case data in basedata.json. Will ensure it works in staging and then launch to prod.

jzohrab commented 4 years ago

timeseries.csv crashes now ... https://api.staging.covidatlas.com/reports/status?format=html

In staging, we hit the max memory for the lambda (3008 MB!!!).

https://us-west-1.console.aws.amazon.com/cloudwatch/home?region=us-west-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252FLiStaging-ReportsV1-1H1EAZF6H1PZ/log-events/2020$252F08$252F03$252F$255B$2524LATEST$255D240a20e449034368b8ea32d92bace91e$3Fstart$3DPT3H

This won't affect the FE, but it will affect anyone using that report.