SeattleTestbed / utf

Unit Test Framework for SeattleTestbed
MIT License
1 stars 9 forks source link

Tests should stop if `subprocess` or `setup` scripts fail #53

Open aaaaalbert opened 10 years ago

aaaaalbert commented 10 years ago

When running unit tests that employ a ut_MODULE_subprocess.py or ut_MODULE_setup.py script, an error in these scripts does not hinder utf.py to try to run other test files following the assumed initialization / setup. In the "best" case, the subsequent tests will fail and report so, in the worst case, the tests will wait for a precondition to be fulfilled by init / setup, and thus hang indefinitely. (See SeattleTestbed/seash#80 for an example of the latter).

utf should use subprocess.communicate() to check for errors in any processes it spawns, and stop if errors are encountered. Similarly, any ut_MODULE_subprocess scripts need to check for (and raise) errors of ut_MODULE_setup scripts they spawn.

aaaaalbert commented 9 years ago

Unfortunately, Windows doesn't seem to spawn the subprocess unless we close stdin / stdout. This means we can't trivially monitor the subprocess for errors. See SeattleTestbed/seash@26cf2702be2872810b4d60f9e3d43837d14db054 that attempted to introduce monitoring to a seash unit test, and SeattleTestbed/seash@37637ba1cebe1a98481ed4a2ad668a771fea9757 that removed it subsequently due to the Windows issue described above.