google / mobly

E2E test framework for tests with complex environment requirements.
https://github.com/google/mobly
Apache License 2.0
647 stars 179 forks source link

Fix assert message to have a single stacktrace #565

Closed mderu closed 5 years ago

mderu commented 5 years ago

In Python3, the stacktrace is duplicated when an exception is caught and raised as a different Exception object. We can get around this by determining the assertion failed, and re-raising the assertion as a TestFailure.

Also, AssertionError is caught instead of Exception. It is the base class name for the errors pyunit raises in both python2 and python3.


This change is Reviewable

mderu commented 5 years ago

No, it's far less severe than that. When you run this in python2 you get the following:

$ python2
>>> from mobly import asserts
>>> asserts.assert_equal(1, 2, 'msg')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mobly/asserts.py", line 55, in assert_equal
    raise signals.TestFailure(my_msg, extras=extras)
mobly.signals.TestFailure: Details=1 != 2 msg, Extras=None

When you run this in Python3, you get:

$ python3
>>> from mobly import asserts
>>> asserts.assert_equal(1, 2, 'msg')
Traceback (most recent call last):
  File "/usr/local/google/home/markdr/mobly/mobly/asserts.py", line 47, in assert_equal
    _pyunit_proxy.assertEqual(first, second)
  File "/usr/lib/python3.6/unittest/case.py", line 829, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.6/unittest/case.py", line 822, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 1 != 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/google/home/markdr/mobly/mobly/asserts.py", line 57, in assert_equal
    fail(my_msg, extras=extras)
  File "/usr/local/google/home/markdr/mobly/mobly/asserts.py", line 239, in fail
    raise signals.TestFailure(msg, extras)
mobly.signals.TestFailure: Details=1 != 2 msg, Extras=None

With this change, the stacktrace is simplified to more closely represent what it looks like in pyunit:

markdr@markdr:~/mobly$ python3
>>> from mobly import asserts
>>> asserts.assert_equal(1, 2, 'msg')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/google/home/markdr/mobly/mobly/asserts.py", line 55, in assert_equal
    raise signals.TestFailure(my_msg, extras=extras)
mobly.signals.TestFailure: Details=1 != 2 msg, Extras=None

Because the framework intends to make the assert functions appear to be native, it makes more sense to hide the fact that we are raising an AssertionError with pyunit, and re-raising it as a test failure. End-users shouldn't be exposed to this quirk, which ultimately makes the stacktrace in Python3 more confusing.