report unhandled exception in testing framework

[e3krisztian]

Fixes #60 - uncaught exception from testcase - exit without traceback

[e3krisztian]

I have seen g, but did not know what it is for, installed green in two separate virtualenvs, e2 for python2 and e3 for python3:

for v in 2 3
do
    virtualenv$v e$v
    . e$v/bin/activate
    # -e . "installs" current directory as "editable" package - future edits will be immediately available
    pip install -e .
    pip install mock pytest testtools coverage
    deactivate
done

then I could run e2/bin/green and e3/bin/green

[e3krisztian]

Multiprocessing is a great feature with current machines!

This PR is too simple to work in a multiprocessing runner. When I saw the code I was modifying it was obvious that more work would be required, like using the stream object for color output or passing the info back to the parent runner and also a test against regression.

[CleanCut]

Okay, I went quite a bit deeper into the code and handled stuff in the poolRunner function. I think this solves your problem in a similar way to what you were proposing, and also goes well with the worker initializer/finalizer work I'm working on in a different branch.

If you want to try it out yourself right now, it's in this branch: https://github.com/CleanCut/green/tree/catch-unhandled-exceptions

Otherwise, you will be able to try it out when I release version 1.10.0, hopefully tomorrow sometime.

[e3krisztian]

The exception handling looks fine to me! :)

Some thoughts after reading the code:

• I see that you have some tests for handling SIGINT, but have you ever tried to stop long-running multiprocessing tests with ctrl-C (raises KeyboardException) on python2? there is a known issue with multiprocessing and ctrl-C:
  • https://github.com/jreese/multiprocessing-keyboardinterrupt (check out the issue and pull request as well)
  • http://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool
• I have done similar tests than in green/test/test_runner.py: enter temporary directory, write some file, process the file, restore original context. I use a context manager to do the chdir and back. Your approach with setUp and tearDown is interesting and new to me.
  • I fear using tearDown - is this this issue applies?
  • you might consider using the more direct addCleanup instead of tearDown if you do not care about supporting 2.6.
• there is some boilerplate code when creating text files and source formatting looks strange with unindented multi-line string
  • you might replace the repeated open, write(content), close sequences with a write_file(path, content) function
  • a similar function write_text_file(path, content) could call textwrap.dedent() for the textual content to remove whitespace prefix from multiline string constants

[CleanCut]

(I wish github had a "quote" feature).

• Regarding Ctrl-C/KeyboardInterrupt: Yes! I have worked extensively to handle Ctrl-C as elegantly as I can. I have been able to make it so that about 95+% of the time hitting Ctrl-C will be nicely caught and cause the remaining tests to be aborted and we report on the statistics so far after printing out a message that some tests may not have been run. The remaining ~5% of the time, the code is in some spot that I can't figure out how to catch, so the process just crashes. Either way, the testing stops. Try it out. It took me weeks to get this functionality figured out to the level it is today.
• Regarding tearDown -- the issue you linked to affected python 2.4-2.6 and was closed in favor of another issue that was implemented circa 2010. Since green only supports 2.7 and 3.4+, no I don't believe that issue is a concern.
• Regarding addCleanup: I agree. I should refactor some of my own tests to make them less boilerplate-y.
• Regarding boilerplate-optimization: I agree here too. Thanks for pointing out textwrap.dedent() -- I had seen that used by someone long ago, but I was not aware it was a built-in as of 2.7.

I created #62 to make sure I don't forget.