Approach and API spec

[geohacker]

We'll build the database and API using an array of tools like ogr2ogr for ETL and FastAPI / pygeoapi for querying. After diving into the use cases a bit more and comparing various approaches, we think Hecate is not a good fit at the moment.

Goals

1. Geo data — import admin and fishnet layers with unique ids that can be matched to different indicator layers.
2. Layers — introduce custom schema for different layers of data. We should account for being able to add new data layers.
3. Catalogue — one of the goals is to allow people to discover indicators and thematic groups of datasets.
4. Flexible schema, indexes and caching — there are lot of indicators and datasets that we will load initially and this needs a flexible schema that we can design for better query performance. We will also need to think about caching as the number of requests increase.
5. Support for time series data — even though time series isn’t a priority right now, the nature of a lot of the indicators are time series. The dashboard will want to query these.

Data

• Currently all datasets are shared via OneDrive. This is proving to be slightly tricky for programmatic download and also receiving updates. Could GOST provide an archive of data that we can download programmatically (using wget/curl), from S3 or OneDrive directly? @KPGeo

Data ingestion

• PostgreSQL + PostGIS
• ETL scripts using combination of bash, command line OGR, and possibly some python. Once we have the datasets we need to pull in, OGR can create the table structure that we need.

API

This is the first draft of the API spec. We'll finalize this as the data is imported.

Catalogue

• GET /api/\/catalogue -- returns a generic JSON with all the data layers available in the database
• GET /api/\/risk-schema -- return the risk schema JSON. I realize this is different from the catalogue and may not contain all the datasets available.

Geo

• GET /api/\/geo/list -- list available categories - admin0, admin1, admin2, urban, hd_urban. Can omit in case the catalogue is enough
• GET /api/\/geo/list?country= -- returns a list of all the geo layers with names for the specified country
• GET /api/\/geo/geojson?country= -- returns a list of all the geo layers with names for the specified country

Layers

• GET /api/\/layers/list -- lists all available layers. Perhaps similar to the catalogue JSON with sources and other metadata. Can omit in case the catalogue is enough
• GET /api/\/layers/list?country= -- lists all available layers and resolutions for a given country.
• GET /api/\/layers/geojson?country=&?geo= -- returns GeoJSON of the layer at that resolution
• GET /api/\/layers/csv?country=&?geo= -- returns CSV of the layer at that resolution

@guidorice @bitner @pieschker @KPGeo

[KPGeo]

@geohacker will ask Dany about the OneDrive / S3 access. I think S3 has some institutional lock down issues, but it can't hurt to ask. We've been using direct access to OneDrive through file explorer and updating the registry to enable larger file movement. More to come.

[geohacker]

Some updates here - instead of writing custom endpoints, we are using a combination of FastAPI and PygeoAPI. Both projects have great community behind it. We have used FastAPI for a lot of other projects at Devseed and think it'll be great choice.

To see how this will work, we have a staging stack running here http://covid-publi-1v66das8fk57r-771481456.us-east-1.elb.amazonaws.com/docs. This is a FastAPI stack that wrap around Pygeoapi routes.

We are also working on implementing an api key using OpenAPI api key and http basic auth schemes. More on this next week!