badpandabear
commented
4 years ago With the latest set of changes I split the pypeit_test file into
- test_main.py: The frame work for running tests
- pypeit_tests.py: The classes used to run different test scripts
- test_setups.py: The definition for what setups and instruments are part of the dev suite, and which tests to run on what setups.
As discussed in the slack, I also added a test_priority_list which contains the test setups in the dev suite in decreasing order of duration. It is re-written every time a full run of the dev suite finishes, if anything has changed. This replaces the earlier TestLength logic.
profxj
This is a large set of changes to run the dev suite test setups in parallel, add some reporting options, and allow for pytest testing of the pypeit test script.
The pypeit_test script was split with most of the code going into test_scripts/pypeit_test.py. This allowed for unit tests in test_pypeit_test.py to import the main function and to test it with mocked subprocess code. The pytest tests take two minutes to get 96% coverage of pypeit_test.py, instead of taking hours to discover a syntax error at the very end of a test run.
Test setups can be run in parallel from a PriorityQueue. The test setup was encapsulated into a TestSetup class that can categorizing setups based on how fast they run. The slowest tests are run first for efficient CPU utilization. If the slower tests were run later in the test run they could leave CPUs idle once the shorter tests have completed. The tests are roughly categorized into SHORT, MEDIUM, LONG, VERY_LONG, and EXTREMELY_LONG categories.
Test types have been encapsulated into a class hierarchy with common code to run tests in the base PypeItTest class.
Reporting logic is separated out into a TestReport class that reports on testing as it runs and can also generate test reports.
There is a -r option now for writing a detailed test report to a file. The detailed report can also be printed to stdout if the -v (verbose) option is given.
There is a -q option to suppress all output to stdout so that testing can be run in the background with a detailed being written to a file.
Including a detailed test report and the default stdout from a test run.
pype_test_output.txt
two_threads_sep11.report.txt