Scope test coverage and strategy

[billgeo]

Task

The gazetteer has no automated testing. For the health of the product and to allow us to make changes we need automated testing.

To do this we need to know how much testing is needed, high level view of how we will test and also consider any migration that will be necessary when migration the plugin from QGIS 2 -> 3

Definition of Done :

• [x] We have a clear and agreed test strategy
• [x] We have an idea of the test coverage we need
• [x] We have considered any migration of tests when moving from QGIS 2 to QGIS 3

Out of Scope

Discussion required

• Needs discussion with architects and other QGIS plugin devs

Add an Assignee, Epics, Estimate and any relevant Labels.

[billgeo]

We were thinking end to end testing of QGIS UI, using containerization, but maybe this is overkill? Looking at other LINZ QGIS plugins it seems that no one else is doing this.

This stack exchange post seems to cover how to do testing in QGIS container. Unfortunately the Boundless blog post on this is a broken link since they were bought out by Planet software. https://gis.stackexchange.com/questions/71206/writing-automated-tests-for-qgis-plugins

Would appreciate any help, discussion or links to best practice etc @dwsilk @imincik @palmerj

[dwsilk]

See (perhaps poorly named) unittest step here: https://github.com/linz/nz-buildings/blob/master/.github/workflows/build.yml

This uses docker-compose, with a PostgreSQL/PostGIS container and a QGIS container, and the test_runner that was originally developed by Boundless in your link above.

Official QGIS docker containers contain test helper scripts that were originally created and maintained by Boundless.

[billgeo]

Thanks @dwsilk . We will do something similar.

[billgeo]

Decisions:

• Test strategy is do end to end testing using docker containers and QGIS UI testing tools.
• Test coverage will be minimal to ensure correct function of the plugin and ensure data integrity.
• Considered migration of tests from QGIS 2-> 3 (plus python 2 -> 3 and QT 4 -> 5). Agreed that it will be very time consuming. Easily more time porting the tests than the source code. It will probably be more efficient to do manual testing for port of plugin from QGIS 2 -> 3. And can then write automated tests based for QGIS 3 version of plugin. To be confirmed once team have finalised a test plan.

[palmerj]

@billgeo I would suggest the following:

1. Unit testing for the database components and the QGIS UI. This step will need mock input data or interfaces.
2. Integration testing with both the database and the QGIS UI. This can contain your end-to-end test cases in it.

Note if timeframes are a problem you can drop 1/, but my feeling is that will help with the development of the component with CI. e.g setup of docker containers and running tests for those containers.

I think we should use pytest (rather than unittest) and pgTap for the database and python testing frameworks.

[billgeo]

Thanks @palmerj seems to be the consensus to some DB testing. @strk also suggested testing the schema (at least) in a PR elsewhere (that I now can't find).

[palmerj]

Cool. @strk comment is here https://github.com/linz/gazetteer/issues/62#issuecomment-597682363

[dwsilk]

I think we should use pytest (rather than unittest)

I think this would require a number of changes to the way testing is handled by the official QGIS docker containers, but I'd like to use pytest too, so if it can be done..

[palmerj]

I think this would require a number of changes to the way testing is handled by the official QGIS docker containers, but I'd like to use pytest too, so if it can be done..

Hmm.. What's the level of changes needed?

[billgeo]

Done. We have:

1. test plans from NZGB
2. An environment and automation for tests
3. Tested automation with some basic QGIS UI tests