Functional Testing in Robosat

[bkowshik]

There are couple of open PRs in Robosat that affect multiple files and folders. I am thinking having a system test in place gives us more confidence in rebase and merging these PRs. Having this run for every commit on Travis would be amazing!

Steps

The different steps in the Robosat pipeline can be grouped into the following 4 groups. Grouping makes thinking about writing tests a little easier with clarity of what are inputs and outputs for each group and for a lot of cases, the output of the previous step is used as input for the next step.

Group 1. Data preparation

This group begins with a pbf file. So, ideally we create a small pbf file representing it and store it in the repository itself so that all tests are repeatable with examples based on this testing dataset.

• extract: Extract parking lots from OpenStreetMap pbf files as geojson
• cover: Cover parking lot geojson with zoom 18 tiles and output a csv
• rasterize: Burn parking lot features from geojson into tiles from the csv
• weights: Calculate weights for sample dataset
• download: Download Mapbox satellite imagery using Maps API

Group 2. Model training

Since, we want these tests to run on CPUs from TravisCI, the current strategy would be to keep the numbers of samples in the training and validation dataset small, 25-50 samples maybe.

• train: Train a machine learning model
• predict: Get predictions from trained model

Group 3. Post-processing

Need to make sure that the samples in the testing dataset as split across multiple tiles so that we can test the behavior of the merge step. Similarly, for the testing dataset, we could draw out a few polygons to represent parking lots mapped in OpenStreetMap instead of using raw OpenStreetMap data.

• mask: Convert per class probabilities into per pixel background/parking labels
• features: Convert from pixels to geojson features
• merge: Merge features across tile boundaries into one
• dedupe: De-duplicate with already mapped features on OpenStreetMap

Group 4. Infrastructure

Tests for this group are of a lesser priority in comparison to the above three steps. So, this can definitely come later once we have the system tests for the above three groups finalized.

• export: Export models into ONNX format
• serve: Get model predictions in real-time
• compare: Tool to merge images, masks and predictions
• subset: Tool to prepare a subset from a slippy map directory

[bkowshik]

Spent an hour researching into System Testing in general and following are my notes:

• System Testing is testing of a complete and fully integrated software product end-to-end; like the way a user would use the system.
• System Testing can be categorized as Black-Box testing; functionality of the application is tested without looking into its internal structure. The key is preparing inputs and the expected outputs.
• There are ~15 types of System Testing that include both functional and non-functional. Ex: Stress, Regression, Functional, Usability, Security. This ticket will focus on Functional Testing; usually describing what the system does.

Hyperlinks

• https://www.guru99.com/system-testing.html
• https://www.youtube.com/watch?v=3lOjtmsA2Kk
• https://en.wikipedia.org/wiki/Functional_testing

Next actions

• [x] Rename ticket heading to Functional Testing instead of System Testing

[bkowshik]

Looked around for examples for system tests, particularly in Python; not sure if I am looking for right things. Would love to :eyes: some example projects with functional tests.

Tests for Robosat cli

Taking inspiration from how Mercantile has tests for it's cli, we could have a similar setup where we have a tests/test_cli.py file that tests the different steps in the Robosat pipeline like rs extract, rs cover etc. Something I want to try out is when a tests file has all the test function laid out, will the test runner run them in order or is that not predictable. If the tests are run in order, we could potentially wire-up in such a way that outputs of one step become the inputs to the next step. 😃

• https://github.com/mapbox/mercantile/blob/master/tests/test_cli.py

[jqtrde]

Think this is stale - going to close but please yell if that's not the case!