Present options for making data available to tests

[SPlanzer]

Task

We need data for running tests. How are going to get this into the test database (db container) for running automated tests?

Definition of Done

Once there is agreement on the best approach for making data available to automate tests,

Out of Scope

Discussion required

Yip. Input as to the best method for making test data available to automated tests.

Add an Assignee, Epics, Estimate and any relevant Labels.

[SPlanzer]

Options

1. Full dump and restore from production
2. Partial dump and restore from production
3. Create required data programmatically on the fly prior to test execution

1. Full dump and restore from production

Bad

• Slow - production is 1.3gb in plain text data. This is a lot of data to restore into the DB for tests and will slow done deployment/tests

Good

• integrity - its real prd data.
• Useful for UAT if UAT tester can use docker

Summary

This is going to be slow. Either when building the docker db container or at test initiation a large amount of data will need to be read into the DB. - This is going to be slow for our needs and slow down development

2. Partial dump and restore from production

Bad

• complex - How do we find the data we require that covers all test scenarios and maintain referential integrity when selected data across tables.

Good

• integrity - its real prd data.

Summary

Cherry picking a subset of the production data may be complex. It will not be overly difficult to ensure data covers all test scenarios but it may be hard to maintain referential integrity across tables.

3. Create required data programmatically on the fly prior to test execution

Bad

• integrity - its not real prd data.

Good

• Tailored - it can be tailored to ensure there is data available for all test scenarios
• Fast / only the required data is added to the DB

Summary

Creating test data on-the-fly is the most flexible. It allows for us to have the required data available and deployed quicker than other methods.

The gazetteer feature data can be created via the database applications stored procedures to ensure referential integrity is maintained.

example of creating test data

import psycopg2
from psycopg2 import Error

test_data = [
    {
        "name": "A Hill",
        "code": "PLAC",
        "geom": "POINT(174.85193754181844 -37.796524856134)",
    },
    {
        "name": "A Islands",
        "code": "ISLD",
        "geom": "POINT(174.85193754181844 -37.796524856134)",
    },
    {
        "name": "A Lake",
        "code": "LAKE",
        "geom": "POINT(174.85193754181844 -37.796524856134)",
    },
    {
        "name": "A Glacier",
        "code": "GLCR",
        "geom": "POINT(174.85193754181844 -37.796524856134)",
    },
]

# Connect to DB
try:
    ps_connection = psycopg2.connect(
        user="gazadmin",
        password="gazadmin",
        host="127.0.0.1",
        port="5432",
        database="gazetteer",
    )

    # Get cursor
    cursor = ps_connection.cursor()

    for data in test_data:
        # Run database applications gaz_createnewfeature procedure
        cursor.callproc(
            "gazetteer.gaz_createnewfeature",
            [data["name"], data["code"], data["geom"]],
        )

    # Commit
    ps_connection.commit()
    cursor.close()
    ps_connection.close()

except (Exception, psycopg2.DatabaseError) as error:
    print("Error while adding a new feature to the database", error)

[billgeo]

Sounds like a no-brainer to me. I'm good with dummy test data.

[SPlanzer]

I agree with Bill. Seems we have consensus - closing