Data API for powering visualisations like the map

[rkassa]

Current Behavior

There are scheduled "cron jobs" on Data that export country and regional case (and VOC) data as a JSON object. This JSON object is dumped into an S3 bucket and is then used by the Map visualization. There are a few issues here:

• The data displayed on Maps is never really actually "current/live" — it's as old as the last scheduled JSON dump.
• Maps performance will certainly be compromised as the JSON dump gets larger; we should be requesting for and using data points only as needed -There is no elegant way of including demographic and metadata without adding further bloat to the JSON dumps

New/Desired Behavior

We would have API endpoints for Data instead of running scheduled JSON dumps from the database. These endpoints with parameters would look something like this:

• Countries List: https://data.covid-19.global.health/api/list/countries
• Regions List: https://data.covid-19.global.health/api/list/regions
• Case Counts by Country: https://data.covid-19.global.health/api/casecounts?by=country
• Case Country by Region: https://data.covid-19.global.health/api/casecounts?by=region
• Country Detail: https://data.covid-19.global.health/api/casedetails?country=US
• Region Detail: https://data.covid-19.global.health/api/casedetails?country=US&region=Alaska
• Filtered Cases: https://data.covid-19.global.health/api/cases?gender=F&country=US&travel=TRUE
• And several more :)

This way, the line list data (with all demographic data) can be usable, with filters, on the Map visualization. It would scale significantly better than continuing to accommodate large JSON data dumps as well. We will need to create more granular issue tickets to scope this out better.

[abhidg]

@rkassa The database is updated twice daily, so actually the JSON dumps can be made less frequent (currently, it runs 4 times a day).

Would the issue of map performance be fixed by adding granular JSON dumps? Then map could request only the ones that are needed.

• List of countries and regions - this wouldn’t save time compared to the current dump
• Case counts by region could be useful to have an API for
• What does /api/casedetails return?
• Filtered cases should work now

[joe-brilliant]

Re: #2142 (cc @iamleeg @rkassa @jim-sheldon )

[iamleeg]

@rkassa I'd be very interested in getting this API built out, not only would it improve the maps application (it'd definitely help with showing dev data in dev, prod data in prod, as well as improving the liveness of the data and shrinking the size of the browser download) but also improve the lives of integrators who want to mix our data with other sources.

I think you're right that we need one ticket per API endpoint, then we can take an incremental approach where each use of the JSON files replaced until eventually we don't need to distribute the JSON at all and can turn them off. For each of the endpoints you list it would be great to know:

• how this would be called in the map (e.g. query parameters, expected responses)
• what does it replace from the JSON, so we can see more detail on what data is needed
• where in the map it gets used, so we can test that the replacement works the same way

For whichever one of these we build first there will be cross-cutting concerns that we need to think about to apply to all the others:

• will we need rate limiting on these endpoints?
• should there be server-side authentication, perhaps through static API keys?
• how will the client react to the API being unavailable?
• what mongo indexes do we need to efficiently serve these APIs from "live" data?
• do we calculate responses live, or precalculate them maybe during data ingestion?