Develop method for API data sources

[kzollove]

NOTE: This issue should be refactored before work begins. Example datasets/API calls: SDG Indicator database ACS 5 Year (site) USGS Daily Values (site) India government API (site)

[kzollove]

https://api.dhsprogram.com/#/index.html

[kzollove]

Some starting points/ thoughts for defining this method/best practice:

Best practices for granularity:

Spatially - a data source should not be subdivided past the level of an entire country

Temporally - a data source should not be subdivided past the level of an entire year

Examples:

• Census TIGER/Line us-state-2014; This dataset has census polygons of every US state for the year 2014. It is is easy to use at this granularity, and an end user can specific states if they need to. It could not be broken down further
• EPA AQS daily_aqi_by_county_2020; This dataset has attribute data for US counties for the year 2020. If an end user only needs a specific county or counties, they can filter for that on their end. While the data within the dataset are tied to specific days, all the data for the year should be downloaded at once. It is up to the end user as to how they want to filter, aggregate, or otherwise use the data.
• City of Cambridge MA GIS HEALTH_hospitals; This dataset has polygons and addresses for hospitals in Cambridge MA. Its obviously far below country level spatially, and has no temporal component (besides date it was last modified). Since there is no way to "upscale" the dataset to nationwide coverage, it is perfect as it is.
• DHS Open Indicator API, countryIds=NG and surveyYear=2010 and format=csv; This dataset typically includes all fields for all countries for all years with a format of JSON. This is the ideal API for our software to interact with: it allows us to specify a country and year, and even returns data in CSV format.

The DHS API has a bunch of other fields on which to filter. I think we need to play around with the country-year level first to see the file size range of these datasets, but I am hopeful we can give a hard stop at the country-year line for data sources. All of those fields that are returned in the dataset are potential variable sources, as I see it.

Another thing moving forward, should we restrict our scope of potential API sources to those that offer CSV output? Just at the beginning while we gather our bearings, maybe.

[kzollove]

@tibbben would love to get your insight on this. @rtmill and I started a conversation with a big group last week, but it ended pretty pessimistically. We know you are already implementing API downloads. We are cautiously hopeful that we can develop a slightly more dynamic (less hardcoded) solution if we try really hard. That said, your work would be a really solid starting point as we are currently shooting in the dark. Maybe a meeting for the three of us in the next week or two?