search

VREMSoftwareDevelopment / WiFiAnalyzer

Android application to analyze WiFi signals.
https://vremsoftwaredevelopment.github.io/WiFiAnalyzer/
GNU General Public License v3.0
3.24k stars 606 forks source link

We found 1 test flaky #298

Closed shaker-project closed 4 years ago

shaker-project commented 4 years ago

We run your test cases for 50 times and found that the following test is flaky [1] (i.e., they pass or fail non-deterministically):

We ran these tests on a generic Android Emulator (AVD) with Android API version 28.

To reproduce failures on these tests, please run the following script on gist: https://gist.github.com/shaker-project/aa36af5d1be2638caf8f6e2e50768864 If you didn't understand any point of the script we can explain.

We require that stress-ng is installed. If not, run $> sudo apt-get install stress-ng.

If you confirm this behavior, our suggestion is to add the annotation @TestFlaky on these tests. See https://developer.android.com/reference/androidx/test/filters/FlakyTest

If you wish for a repair proposal, please let us know.

[1] https://hackernoon.com/flaky-tests-a-war-that-never-ends-9aa32fdef359

VREMSoftwareDevelopment commented 4 years ago

Can you please explain what fails in the above test testNavigation, since it does not have any assertions?

shaker-project commented 4 years ago

We apologize for the delay in responding.

Well, the exception raised was NoMatchingViewException. We investigated that, the test fails on line 82 of the selectMenuItem method when it is called selectMenuItem(SETTINGS) on line 59 of the Navigation class. We believe that this happens because the rendering of the element can be slow, therefore, making Espresso not find the element.

The file: output_wifi_testNavigation.txt, contains the test execution outuput.