switch to a testing framework that supports parallel execution

[stevekm]

Right now we've been using the Makefile with make test3 -j16, etc., to run test cases in parallel since unittest does not support parallel execution. Consider switching to pytest with its plugin for parallel execution

https://docs.pytest.org/en/latest/ https://pypi.org/project/pytest-xdist/

[stevekm]

also the official Python dev guide here https://devguide.python.org/ suggests that there is a way to run tests in parallel;

./python -m test -j3

need to figure out what this is referring to, in case newer versions of Python have built in parallel test execution?

[stevekm]

Another suggestion for this, is to use external processes to run specific required tests in parallel

For example, a specific test can be run with e.g.

$ LARGE_TESTS=True python tests/test_workflow_with_facets.py TestWorkflowWithFacets.test_run_worflow_two_mafs

so we can also just use any basic parallel task execution method to run multiple specific tests in parallel like this

[stevekm]

switching this to high priority because the current test suite takes so long to finish that its unfeasible to use, I left it run overnight with LARGE_TESTS=True INTEGRATION_TESTS=True CWL_ENGINE=Toil USE_LSF=True make test3 -j 30 and it still took about 12 hours to finish.

Also a major issue is that while using the make test3 -j n command, if one test case breaks it halts execution of any new test cases, so its impossible to finish the test suite if any test fails unless it was launched prior to the failure.

[stevekm]

implemented with the parallel-test Makefile recipe

https://github.com/mskcc/pluto-cwl/blob/32f869aedad4b70574b9dff966fd18f775adc85e/Makefile#L256-L277