Develop a Test Asset -> Test Case script

[maximusunc]

This script should be able to take the test assets and create test cases, and potentially test suites.

The tentative test asset input is a TSV file.

[RichardBruskiewich]

Is this need partially covered by the proposed "Test Documentation Application" which creates the specifications for the above in the test suite repository, either as static Test Case/Suite instances (i.e. explicitly enumerated in the repo) or as a dynamic specification that assumes a Test Harness and/or Runner dynamically generating instances of Test Case/Suite (i.e. like SRI_Testing-like runners), with the test suite repo accessed using an OpenAPI interface?

[maximusunc]

My understanding is that the "Test Documentation Application" is for the ingestion of data in order to create valid Test Assets. This issue is for creating a script that converts Test Assets into Test Cases and subsequent Test Suites. I imagine this script could live in the Tests repo, so no need for any API.

[RichardBruskiewich]

My understanding is that the "Test Documentation Application" is for the ingestion of data in order to create valid Test Assets. This issue is for creating a script that converts Test Assets into Test Cases and subsequent Test Suites. I imagine this script could live in the Tests repo, so no need for any API.

Hi @maximusunc, it's a bit of a grey area. The "Test Documentation Application" is meant to capture all aspects of tests: Metadata, Assets, Cases, Suites - as required. In some cases - e.g. SRI_Testing-like Test Runners - the TestCase and TestSuite are dynamically constructed based on a specific kind of TestAsset, hence may not (but could be) archived in the test suite repo as well. Like I said, this is a grey area since maybe this is unnecessary if the Test report database tracks things sufficiently.

As we've already noted, "one shoe fits all" doesn't really apply to Translator testing: there are various scenarios and testing objectives to consider. I'm sure we'll iterate a bit on this over time.

[maximusunc]

Maybe we should rename this ticket to just be "Develop the ways that Test Assets, Test Cases, and Test Suites get created"? That's all I'm really trying to get at.

[sierra-moxon]

This is the current script: https://github.com/TranslatorSRI/TranslatorTestingModel/blob/main/src/translator_testing_model/scripts/generate_suite_for_demo.py

[RichardBruskiewich]

Good afternoon @sierra-moxon and @maximusunc,

You (plus Andrew) will have noted my comments on Slack about TestSuite access.

I am essentially proposing to code a (relatively simple) Python library API layer on top of the autogenerated model code, which will broker the traffic in between the Tests and the TestHarness, perhaps by means of some iterator/generator method class wrapping all the data sources.

Yes... I understand that the TestHarness is a kind of broker of the TestCase entries as well, so one might be tempted to place the above code there; however, I was thinking that since the landscape of TestAsset sources and related metadata is likely to be somewhat fluid for awhile, that we may wish to keep the above access code close to the model repo.

[maximusunc]

Done!