utf.py does not run shutdown/subprocess/setup scripts without base test

SeattleTestbed / utf

Unit Test Framework for SeattleTestbed

MIT License

1 stars 9 forks source link

utf.py does not run shutdown/subprocess/setup scripts without base test #48

Closed choksi81 closed 8 years ago

choksi81 commented 10 years ago

As of the current revision (r7170), the ut_utftests_test_setup.py, ut_utftests_test_shutdown.py, ut_utftests_test_subprocess.py the utftests suite do not run.

Unit tests in the form of ut_modulename_testname_shutdown.py, ut_modulename_testname_subprocess.py, ut_modulename_testname_shutdown.py do not get run if there is no test named ut_modulename_testname. When ut_modulename_testname.py exists, then the shutdown/subprocess/shutdown scripts run normally.

choksi81 commented 10 years ago

Author: leonwlaw The line that is responsible for this behavior is line 400 in utf.py. It removes the test script if it ends in *_(setup|shutdown|subprocess).py. The logic is sound iff there is a base script there. Therefore, files that have those endings that don't have a base test file (and hence, are not actually companion unit tests) do not get run at all.

A simple fix is to check if there is a corresponding base test before removing the unit test from the list of tests to run.

choksi81 commented 10 years ago

Author: leonwlaw My patch adds 2 unit tests:

ut_utftests_nobasetest.py: Tests running unit tests that don't have a base test. (The problem in this ticket)
ut_utftests_test_companion_runcount.py: Makes sure we run each setup/shutdown/subprocess test the correct # of times. My initial trivial patch of simply removing utf's line 400 produced a case where the setup test was being run twice. The patch can be found here:
https://seattle.poly.edu/attachment/ticket/1384/tests.2.patch

choksi81 commented 10 years ago

Author: leonwlaw I have changed the comments slightly on utf.py. The patch can be found here: https://seattle.poly.edu/attachment/ticket/1384/tests.3.patch

choksi81 commented 10 years ago

Author: leonwlaw Refactored logic to determine companion tests.

https://seattle.poly.edu/attachment/ticket/1384/tests.4.patch

choksi81 commented 10 years ago

Author: leonwlaw Fixed with r7178

choksi81 commented 10 years ago

Author: leonwlaw This fix introduces module-level companion scripts to run twice, once as companion scripts, and another time as normal scripts. This causes unit test suites that contain subprocess scripts to run indefinitely, as there is nothing to stop them.

choksi81 commented 10 years ago

Author: leonwlaw Consequently, another side effect is that the -f option is ignored completely. This appears to be due to the re-globbing on line 389 of utf.py. The filtering that is done by the caller of the test_file() function is ignored completely. I will have a patch to address this problem and a unit test.

choksi81 commented 10 years ago

Author: leonwlaw My patch to solve this problem is: https://seattle.poly.edu/attachment/ticket/1384/utf.patch. Monzur, could you please perform a review?

aaaaalbert commented 8 years ago

The original issue description has been extended to read

This has been committed in r7364.

Testing with the current utf.py I cannot replicate the problem. Running a test module without a base test still runs the setup/subprocess/shutdown scripts fine. --- Closing!