search

google / mobly

E2E test framework for tests with complex environment requirements.
https://github.com/google/mobly
Apache License 2.0
647 stars 179 forks source link

Improvement: Allow possibility of treating java assertions as test Fails not Errors #528

Open ktkopone opened 6 years ago

ktkopone commented 6 years ago

This may just be a question of me not being able to find it in the docs/examples, but: In our use case we run the bulk of our testing in snippet RPC calls to the device, with java/JUnit assertions being raised within those calls. However, when an assertion fails in the RPC calls, an ApiError is instead propagated up to the python test (seemingly due to the RPC call crashing), and the test thus shows an 'error' rather than a fail. The same holds true for regular java AssertionError.

Since ApiError is propagated up, it isn't ~possible~ easy to use python-side code to distinguish proper test fails from errors, and the test summary will show 'errors' where 'fails' would be more desirable.

It would be nice if there were at least an arg option to make java/junit assertion fails get caught and propagated up through the rpc call into mobly's fail exception type or an assertionerror. This could similarly be done for junit skips.

Sample stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/mobly/base_test.py", line 463, in exec_one_test
    test_method()
  File "app/src/snippet/moblyTests/app_tests.py", line 77, in test_sign_up
    self.deviceA.mbs.signUp()
  File "/usr/local/lib/python3.7/site-packages/mobly/controllers/android_device_lib/jsonrpc_client_base.py", line 336, in rpc_call
    return self._rpc(name, *args)
  File "/usr/local/lib/python3.7/site-packages/mobly/controllers/android_device_lib/jsonrpc_client_base.py", line 307, in _rpc
    raise ApiError(self._ad, result['error'])
mobly.controllers.android_device_lib.jsonrpc_client_base.ApiError: <AndroidDevice|owner> 
-------------- Java Stacktrace ---------------
junit.framework.AssertionFailedError: blah
    at junit.framework.Assert.fail(Assert.java:50)
    at <our package name>.SignUpActivityTest.signUp(SignUpActivityTest.java:76)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.google.android.mobly.snippet.manager.SnippetManager.invoke(SnippetManager.java:166)
    at com.google.android.mobly.snippet.manager.SnippetManager.invoke(SnippetManager.java:90)
    at com.google.android.mobly.snippet.rpc.MethodDescriptor.invoke(MethodDescriptor.java:85)
    at com.google.android.mobly.snippet.rpc.JsonRpcServer.handleRPCConnection(JsonRpcServer.java:118)
    at com.google.android.mobly.snippet.rpc.SimpleServer$ConnectionThread.run(SimpleServer.java:109)
----------------------------------------------
xpconanfan commented 6 years ago

Yeah, we've been thinking about this. It seems reasonable to propagate Junit's assertion errors as Mobly's assert errors. This would require changes on both snippet client and server sides though.

We can give this a try in Mobly 1.9

xpconanfan commented 2 years ago

Sorry this has been pushed back a lot. Because the scope needed to enable this is bigger than expected.

We are considering creating a generic mechanism to propagate errors from snippet server to client side. To make this non-trivial change to snippet lib, we need to first clean up some legacy issues, including the divergence in the implementation of snippet lib for various platforms.