New Test Environment

[ericbusboom]

Create a new test environment.

Kinds of tests.

The test environment will have these kinds of tests:

• Bundle tests, which are the main functional tests.
• Functional tests, to fill in gaps in the Bundle tests.
• Regression tests, to test that bugs are fixed.
• Unit tests, narrow tests of small parts of interfaces.

The focus of testing and coverage will be on the Bundle and Functional tests -- these are the main tests that will be required to pass for releases.

Regression and Unit tests will be primarily used for development.

These tests should be broken up into their own directories:

• test/bundle
• test/functional
• test/unit
• test/regression

The python setup.py test invocation will only run the bundle and function tests. There should be another invocation, python setup.py unitest to run both the unitests and regression tests.

Common Tests setup.

One of the main problems we have now it that there is no common way to setup tests, to set the directory, remote account credentials, etc, required to run all of the tests. In the new model, the environment for running the tests is an Ambry library, even for unit and regression tests. The library is configured based on the user's main ambry.yaml file, by creating a new file from parts of the original one, so the new environment should have a complete, self contained configuration that derives from a parent configuration. Each run of the environment will begin with :

• Reading the current user's .ambry file, following the normal methods of finding it.
• Extracting a file system root from the 'filesystem:' section, and a dsns from the 'database:' section either for a postgres database or a sqlite database.
• Building a new configuration for the tests.

Remotes ( 'library.remote:' config ) are entirely local, and are in the ambry root.

The test-specific config items are:

filesystem:
    test: <root dir for the test library>

database:
    postgres: <DSN of the postgres database>

The DSN for the sqlite database is always 'library.db' in the test root directory.

The python setup.py test and unittest command should have an extra argument for --sqlite and --postgres` These two variants would create a new .ambry.yaml file for the chosen test database, then run the tests.

[ericbusboom]

This feature is on branch newtest#134