Isolate specific job configurations and framework configurations

[trungdong]

Improve the design of the harness framework so that:

• the framework will install/download its default comparator, not the specific job. The job should not have to change the comparator configurations (e.g. path to comparator script) unless it really needs to.
• a specific job will be able to define its converter class(es) and configurations by extending the base classes provided by the framework.

On the second point, the framework currently provides the converter classes for ProvPy, ProvToolbox, ProvStore, and ProvTranslator. I think those should be moved to the corresponding job repos. By so doing, we will also provide examples for converters developed not by Southampton team how to take advantage of the interop test framework without the need to modify the test framework's code.

[mikej888]

I have created package branches of:

• interop-test-harness
• provpy-interop-job
• provstore-interop-job
• provtoolbox-interop-job
• provtranslator-interop-job

The contents of the branches, and how these differ from the original interoperability test harness, are as follows.

Each branch contains a Python package that can be installed using python setup.py install. The respective packages are:

prov_interop
prov_interop_provpy
prov_interop_provstore
prov_interop_provtoolbox
prov_interop_provtranslator

Default test harness configuration files are now located alongside the Python test harness files that read them e.g.

• prov_interop/interop_tests: harness.py, harness.yaml
• prov_interop_provpy/interop_tests: test_provpy.py, provpy.yaml
• prov_interop_provstore/interop_tests: test_provstore.py, provstore.yaml
• prov_interop_provtoolbox/interop_tests: test_provtoolbox, provtoolbox.yaml
• prov_interop_provtranslator/interop_tests: test_provtranslator, provtranslator.yaml

The Python files above look for these files in the same directory as the Python file itself. If a developer wants to use their own configuration file, they can either edit the file, or point to the location of their own one using an environment variable (this latter aspect was already supported in the original design).

When python setup.py install is run, then these configuration files are copied too. So, for example, a developer can use the default configurations without having to set up any local copies of the configuration files:

• prov_interop/interop_tests/harness.yaml specifies a default location for the test cases directory as testcases i.e. a directory in the same directory from which the tests are run.
• prov_interop/interop_tests/harness.yaml assumes that ProvToolbox's provconvert is available on the PATH.
• prov_interop_provpy/interop_tests/provpy.yaml assumes that ProvPy's prov-convert is available on the PATH.
• prov_interop_provtoolbox/interop_tests/provtoolbox.yaml assumes that ProvToolbox's provconvert is available on the PATH.

For prov_interop_provstore tests there is a need for an API key for which, naturally, a default value that works cannot be provided. This test class will use any value in prov_interop_provstore/interop_tests/provstore.yaml unless a PROVSTORE_API_KEY environment variable is defined, in which case that is assumed to hold the API key value. This, again, saves the need to create a local copy of the configuration file.

prov_interop/interop_tests/test_converter.py had a function:

def configure(self, config_key, env_var, default_file_name)

which loaded in a converter-specific test harness configuration file and used it to configure a converter's test class. This has been renamed to:

def get_configuration(self, config_key, env_var, default_file_name)

which returns the configuration. A new function allows sub-classes to configure the converter-specific test class using this configuration.

def configure(config)

This has been introduced to allow developers to write their configuration in-code and configure their test class that way also without needing to use a configuration file at all prov_interop_provpy/interop_tests/test_provpy_hardcoded.py provides an example of an interop test class with hard-coded configuration.

interop-test-harness has scripts/install-prov-interop-comparator, a shell script that uses wget and unzip to download and install ProvToolbox from a ZIP file and update the PATH. This needs to be sourced e.g.

$ source scripts/install-prov-interop-comparator.sh

I'm not sure how ProvToolbox could be installed directly from within the test framework (i.e. within Python) as, while it could be downloaded and unpackaged, the challenge is updating PATH.

Returning to your requests:

the framework will install/download its default comparator, not the specific job.

Not done, beyond provision of a shell script, see above.

The job should not have to change the comparator configurations (e.g. path to comparator script) unless it really needs to.

Done

a specific job will be able to define its converter class(es) and configurations by extending the base classes provided by the framework.

Done

the framework currently provides the converter classes for ProvPy, ProvToolbox, ProvStore, and ProvTranslator. I think those should be moved to the corresponding job repos.

Done

[mikej888]

In the above, the repositories with the converter-specific extensions serve two roles:

• Repository for converter-specific extensions plus unit tests.
• Test runner for interop tests for the associated converter (as embodied in the .travis.yml scripts).

A next step would be to migrate the .travis.yml content that runs the interop tests to yours' and Lucs' ProvPy and ProvToolbox .travis.yml jobs. These would then be responsible for the following:

• Clone testcases.
• Clone interop-test-harness and install package and, for ProvPy, install ProvToolbox binary.
• Clone provpy_interop_job/provtoolbox_interop_job and install converter-specific package.
• Run interop tests.

This would mean that the responsibility for running the interop tests falls to your ProvPy/ProvToolbox repositories (after unit tests have all succeeded, for example), and would be triggered every time ProvPy/ProvToolbox commits are done, which would be more in the spirit of Travis.