Testing Database Management, Take 3

[ericbusboom]

Testing database management is getting complicated and brittle, so it's probably worth actually designing it, rather than letting it evolve.

Requirements

• Test on both Sqlite3 and Postgres
• Separate test environment from production library.
• Database/Library environment is cleaned, reset either between every test class, or every test in a class.
• Should be able to specify common setup conditions without a lot of special code

There are only a few scenarios that differ between Sqlite and Postgres, so most of the testing should be done on Sqlite, with only the different scenarios being tested on Postgres.

Common Initial Conditions

At the start of a test, one of these conditions should be true:

• No library; test will create one
• Entirely new, clean library. ( Possibly cached and copied )
• New library, synced with pre-defined bundles

For postgres, the test should be able to drop and re-create the test database.

Cached library.

A common test need is to have existing bundles in a library. Currently, these bundles are rebuild for every test, or sometimes, for every test class. It would be better to have the tests construct a prototype library, and then copy it into place.

/tmp test structure

The test library can always be in /tmp. It never gets really large. The test tree should look like:

/tmp
    ambry-tests
        proto
        library

At the start of a test, the /tmp/ambry-tests/library directory is and reconstructed. If the test database is sqlite, and a pre-build library is required, it is copied from the proto. If the proto does not exist, it is created.

Configuration

The test configuration requires:

• a test library DSN for the postgres database
• a base file system.

The Sqlite tests don't require a DSN, because it will always be 'library.db' in the root of the library filesystem.

These configuration can be set in a few ways:

• config file
• env vars.

Tests must refuse to run if neither of these configs exist

Config file.

Add a new section of the config file, just for test:

test:
    postgres: < postgres dsn>
    filesystem: <filesystem_path>

This won't be part of the ambry.run config, it's just an extra section only for test.

Env Vars

• AMBRY_TEST_DB. If exists, specifies the postgres DSN. Otherwise, sqlilte.
• AMBRY_TEST_ROOT. If exists, specifies the root of the test file system.

If AMBRY_TEST_ROOT is not set, and there is no test section in the config file, the test setup will immediately fail.

Per test setup

The test class can have set these class-level vars to affect setup. ( Maybe also create class decorators to set them nicely )

• library_init. Can be set to 'drop', or 'clean' to set whether the library should exist at the start of the test. Set to 'class' to init for a specific class, but not between runs. If missing, assume 'class'
• install_bundles. Set to a list of bundles references to sync from the test repository
• init_freq. Set to 'class' or 'test' to recreate the library every test, according to the setting of library_init. If 'class', only re-create once per class. If 'test', do it before every test. If missing, assume 'class'

  Test Init Process

  1. Initialize the library.
     • If library_init = 'drop' or 'clean' completely delete the library directory and re-create it.
     • If library_init = 'class', Check a marker file in the library for the name of the current class. If it is not the name of the current class, delete and rebuild the library. Otherwise, do nothing.
  2. Initialize the database.
     • If library_init = 'drop' and is postgres, drop the library database and re-create it.
     • If library_init = 'clean' and is postgres, clean it ( remove all bundles )
     • If library_init = 'class', Check a marker file in the database ( in a config ) for the name of the current class. If it is not the name of the current class, drop and rebuild the library. Otherwise, do nothing.

[ericbusboom]

@nmb10, could you have a look at this and let me know what you think. Add to any requirements you can think of.