cgiovando
commented
6 years ago To expand on @tyrasd's draft workplan above, I'm going to provide some more details about the first two parts we're working on, specifically gap identification and expanding OSMA coverage.
This work is supported by GFDRR as part of the Open Cities Africa project, a new phase of the Open Cities initiative, set to kickoff in mid-June 2018 across selected cities in Africa.
We envision extending OSMA functionalities to provide project teams in each city the ability to easily analyze map data and the local OSM community dynamics. For example, by adding a gap analysis tool to OSMA, a team can easily identify areas of the city which are missing in OSM, and decide to start a new mapping project.
Working prototypes of the gap analysis tool and expanded OSM types will be available in time by the kickoff meeting in Kampala, the week of June 11-15, 2018. Other functionalities will be added later, focusing first on understanding community dynamics, then other analytical needs as listed in this spreadsheet.
OSM Gap Analysis
Determining completeness of OSM data is challenging and normally requires some external reference datasets to compare to (extrinsic method mentioned by @tyrasd above). From #43 to the recent osma-health implementation of Azavea's OSM vs. WorldPop, the idea has been to use population estimates as a proxy of where OSM buildings should exist, and in what amounts.
Stepping back to just identifying gaps, or simply being able to compare selected OSM features with any external dataset for the same area, we propose an approach based on the current OSMA stack, and in-browser computations between two or more vector-tile values.
This tool should be generic enough to allow for inputing any available dataset (global or local) in raster/grid format (or already in vector tile), and selected specific OSM features/tags combinations to compare to. In-browser (client-side) computations such as simple value difference between vector tiles is achieved with javascript libraries (e.g. http://turfjs.org).
To prototype this tool we plan the following:
- Use the Global Human Settlement Layer (GHSL) global built-up presence, derived from Landsat image collections by JRC, as the external reference dataset;
- Convert the GHSL layer to vector tile format, and aggregate values at each lower zoom level using the same osma-cruncher workflow;
- Add a function/tool in the OSMA interface that allows for comparison of OSM building density with the GHSL layer, and return a color scaled rendering of the results (dynamically).
Extend OSM Data Coverage
At the moment OSMA only provides total counts and density visualization of buildings, and length of roads and rivers. To support analytical needs of the Open Cities Africa projects, and in general, to make it useful to anyone wanting to explore distributions of other OSM features (see #14, #30, #52), we plan to partially refactor the OSMA cruncher and frontend to allow (authorized) users to request any OSM tag combination. For example, in addition to the current building geometry (building=yes) density, we would also like to display only those buildings that have an associated address (addr:street=* plus addr:housenumber=*)
There are limitations and challenges to the planned approach:
- Layers and new tag combinations cannot be requested interactively from frontend-users, as all data in OSMA has to be first processed (daily) by the cruncher;
- The number of OSM tags combinations can potentially be unlimited;
- Management of layers in the OSMA frontend drop-down menu can become problematic when having more than 10-20 layers. This could be improved by thematic groups and expansible sub-menus, or with a search-and-add box;
- If allowing display of more than one layer at a time, some cartographic precautions have to be taken to ensure overlapping density color-scales can be distinguished;
- Not all metadata (e.g. creation time) and elements (e.g. relations) are available in OSMA QA Tiles, the source dataset for OSMA;
To mitigate these issues we plan to set up OSMA with a simple YAML configuration file that will automatically instruct the cruncher about what features to process, and then rebuild the frontend drop-down menu dynamically to show those additional layers.
The YAML file will sit in the public OSMA github repository where users can send their pull requests specifying additional OSM tags to be processed, and then an OSMA instance maintainer will process requests as needed. This workflow is intended to be a temporary solution, before a more structured osma-admin module with a proper user interface and authentication, is developed.
Adding more features to the cruncher daily routine will increase processing time, but with the assumption that buildings and roads (already in the cruncher process) are by far the most used tags in OSM, then we don't expect the process time to increase linearly.
@tyrasd please review and add any technical information that can better explain this work - thanks!
smit1678
awright
tyrasd
→ 
work plan part 1 – priorities and limitations
This describes the possible directions for continuing to go forward in the development of the osm-analytics tool, as well as limitations of the software stack of the existing prototype.
A longer list of features is found in this spreadsheet.
OSMA main principles
osm-analytics turned out to be useful general purpose tool, for example for showcasing OSM mapping activity to the general public amongst other use-cases. Some high level principles made this possible and should (if possible) be kept in future versions of osm-analytics:
priorities for new features
The overall goal in the osm-analytics for OpenCities Africa is to help establish mapping plans (e.g. find out what feature/attributes to map in a specific region). This can be split up into different tasks:
linkage to existing work
work packages
a) gap identification
This is essentially about exploring OSM's data (feature or attribute) completeness. Different methods exist to perform this for a number of specific subsets of OSM features. Extrinsic methods also depend on how the external reference data set is structured (e.g. aggregated data, non-georeferenced data, …) and does in general require a data normalization step (e.g. associating attributes to OSM tags).
We need to narrow down and define the scope (or few number of scopes) that we actually want to cover:
Currently, this is not covered by the osm-analytics framework at all, so both a user interfaces as well as a technical implementation have to be elaborated.
b) community dynamics
osm-analytics already allows limited insight into OSM contribution dynamics by displaying OSM objects' latest edit time and "mapper experience" for a given region.
These features can be expanded to better identify mapping productivity patterns, find out about mapper recruitment and retention frequencies, etc.
c) expanding osm-analytics "coverage"
Improving coverage and accuracy of results generated by osm-analytics is important since the original osma prototype turned out to be limited in a few aspects:
It might be worth rethinking parts of the system design to make osm-analytics future proof.