Automated system testing

[kbernat]

Aim to this topic is to share ideas about how can we integrate automated testing into our Software Development process.

Key points (to be extended or redefined)

• Regression tests
• End to end tests

[JanisGailis]

Do we want to be able to run the tests on all platforms?

If it's enough to test on one platform, I still believe that shell scripts on Linux might be a good way for at least end to end tests.

Another alternative would be to write dedicated python scripts. We could use 'os' calls to test CLI functionality, not just the API.

[forman]

E2E tests should be prepared per use case and include

• One or more IPython notebooks pr stand-alone Python scripts containing Cate API calls
• One or more shell scripts (Win .bat + Unix .sh) containing Cate CLI calls

Not sure how to automate IPython notebook execution maybe there are test runners for this out there?

[JanisGailis]

I think it doesn't make much sense to do them as Jupyter notebooks. I don't see any advantage over plain python scripts there. Jupyter notebooks are nice for presentation, but we don't need that for E2E tests. I can of course convert the demonstration notebooks I make for each UC to a python script.

We could then either make a thin Python main test-runner script, or invoke them from shell scripts.

Ditto for shell scripts. I'm making the CLI demo examples for each UC, so these can be converted into shell scripts.

I assume that we don't validate the output, just test that the script runs and finishes successfully. Is this correct?

[forman]

I move it to IPM5 as I don't see a chance to have it before our release. Note, I started with a scripts directory containing such possible E2E tests for the CLI.

[forman]

@kbernat what is the status here? If any, please report at IRM8.