Replace Datacube Explorer Database Test Infrastructure

[omad]

Continuing on from work started by @Ariana-B in #479 and myself in https://github.com/opendatacube/odc-tools/pull/524https://github.com/opendatacube/odc-tools/pull/524

This PR completely redoes the way database tests work for Datacube Explorer.

Temporary Docker based PostgreSQL server

• If Docker is available, a temporary PostGIS server is run for the duration of the tests. (Pytest Fixture postgresql_server, scope = module)
  • If the database fixture detects it is running within Docker, and that ODC Database access is configured already, it does nothing, allowing the existing testing infrastructure using docker to continue working.
• An ODC Database is created and destroyed per test module, by a new fixture called odc_db of scope module. It creates a new database within the postgres server, named after the test module, and configures ODC to access it via environment variables and temporary configuration files.
• A new fixture called odc_test_db, of module scope, initialises the ODC database structure within the temporary database. It also removes the database after the module is finished.

Test fixture data loading

Previously, test data was loaded a little bit all over the place, using a function which was copied into multiple test modules.

This has been replaced by a new fixture called auto_odc_db of module scope, which looks for module variables (named METADATA_TYPES, PRODUCTS, and DATASETS), and loads them into the current database. This allows each test module to specify exactly which data it expects to be available.

Benefits

In combination, this new setup:

• Allows the running of all the integration tests without any database setup, assuming docker is available.
• Allows running any test individually, with the correct fixtures being used to setup the database and test data.
• Simplifies and makes explicit which test data is used by which test module.
• Makes it possible to run multiple instances of pytest simultaneously (due to unique docker postgres instances), and multiple test modules simultaneously using pytest-xdist, due to unique databases being created per module.

These new fixtures have been implemented in a new cubedash.testutils.database module, which is loaded as a pytest plugin. My intention, once proven, is to move this test module into datacube-core to be available for all downstream ODC based software.

---

:books: Documentation preview :books:: https://datacube-explorer--490.org.readthedocs.build/en/490/

[codecov[bot]]

Codecov Report

Base: 86.60% // Head: 86.04% // Decreases project coverage by -0.56% :warning:

Coverage data is based on head (c0ec914) compared to base (60daf44). Patch coverage: 82.40% of modified lines in pull request are covered.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## develop #490 +/- ## =========================================== - Coverage 86.60% 86.04% -0.57% =========================================== Files 23 24 +1 Lines 3017 3188 +171 =========================================== + Hits 2613 2743 +130 - Misses 404 445 +41 ``` | [Impacted Files](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube) | Coverage Δ | | |---|---|---| | [cubedash/summary/\_summarise.py](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-Y3ViZWRhc2gvc3VtbWFyeS9fc3VtbWFyaXNlLnB5) | `96.05% <ø> (ø)` | | | [cubedash/testutils/database.py](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-Y3ViZWRhc2gvdGVzdHV0aWxzL2RhdGFiYXNlLnB5) | `68.68% <68.68%> (ø)` | | | [cubedash/\_stac.py](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-Y3ViZWRhc2gvX3N0YWMucHk=) | `86.53% <85.26%> (-5.11%)` | :arrow_down: | | [cubedash/\_api.py](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-Y3ViZWRhc2gvX2FwaS5weQ==) | `95.91% <93.33%> (-1.23%)` | :arrow_down: | | [cubedash/\_audit.py](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-Y3ViZWRhc2gvX2F1ZGl0LnB5) | `92.95% <100.00%> (+1.78%)` | :arrow_up: | | [cubedash/\_filters.py](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-Y3ViZWRhc2gvX2ZpbHRlcnMucHk=) | `89.75% <100.00%> (+0.31%)` | :arrow_up: | | [cubedash/\_utils.py](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-Y3ViZWRhc2gvX3V0aWxzLnB5) | `89.18% <100.00%> (+1.98%)` | :arrow_up: | | [cubedash/summary/\_stores.py](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-Y3ViZWRhc2gvc3VtbWFyeS9fc3RvcmVzLnB5) | `92.16% <100.00%> (+0.01%)` | :arrow_up: | | [cubedash/warmup.py](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-Y3ViZWRhc2gvd2FybXVwLnB5) | `65.64% <100.00%> (+0.80%)` | :arrow_up: | | [cubedash/summary/\_schema.py](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube#diff-Y3ViZWRhc2gvc3VtbWFyeS9fc2NoZW1hLnB5) | `80.67% <0.00%> (-0.85%)` | :arrow_down: | | ... and [5 more](https://codecov.io/gh/opendatacube/datacube-explorer/pull/490/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube) | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opendatacube)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[omad]

Switching back to Draft PR.

I haven't explained the testing changes anywhere near well enough, or documented the new pytest fixtures I've added. I just got excited that all the tests were passing.

[pindge]

Please update this too https://github.com/opendatacube/datacube-explorer#contributing-to-integration-test