search

TrueBlueDrive / robotium

Automatically exported from code.google.com/p/robotium
0 stars 0 forks source link

Tests randomly fail for no reason #338

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Create a basic app with ten activities. The activities don't need to do 
anything, just have text.
2. Test case to open and close activities via clickOnText(...), then check 
activity using assertCurrentActivity(...).
3. Run tests a few times.

What is the expected output? What do you see instead?
Expected output should be consistent succeed/fail. My code has a pattern of 
clickOnText(...), assertCurrentActivity(...), goBack(), repeat. Sometimes the 
test succeeds, sometimes it fails, even though I do not change anything, simply 
re-run the test (I mean, really, really, don't change anything, just click the 
run button again).

What version of the product are you using? On what operating system?
Robotium-solo-3.4.1, Intellij IDEA 11.1, Win7

Please provide any additional information below.
The success or failure does not seem to have any reasonable pattern or cause. 
The tests should all succeed, though they randomly fail on the asserts, and not 
every time. It appears that the activity assertion is reading the wrong class? 
I tried using sleep and waitForActivity(), but neither seemed to help.

Original issue reported on code.google.com by Phill.Z...@gmail.com on 11 Oct 2012 at 9:09

GoogleCodeExporter commented 9 years ago 
To clarify (as I am sitting here watching the test run), I notice that the 
proper activity does open and run ( and sometimes even succeeds! ), and that it 
often fails even though it is correct. I triple checked all of my 
activity.class params, and they are correct.

Original comment by Phill.Z...@gmail.com on 11 Oct 2012 at 9:24

GoogleCodeExporter commented 9 years ago 
may be should give solo.sleep() between call and assertion. Sometimes clicks on 
app/Text but before it takes time to open it,assertion will be done.

Original comment by manas...@gmail.com on 12 Oct 2012 at 5:39

GoogleCodeExporter commented 9 years ago 
This actually makes the problem worse. Adding in solo.sleep() between call and 
assertion, with various params between 500 and 5000, causes one of the checks 
(again, a random one each time) to fail for sure.

Original comment by Duno...@gmail.com on 12 Oct 2012 at 5:35

GoogleCodeExporter commented 9 years ago 
Tested a set of tests on a hard device, and every test passed perfected ( I did 
end up having to add in some solo.sleep(...) methods). Ran the same code on an 
emulator, and every single test barfed. Seems to be a problem stemming from the 
emulator interfacing.

Original comment by Phill.Z...@gmail.com on 12 Oct 2012 at 9:27

GoogleCodeExporter commented 9 years ago 
Thanks for reporting this. What version of Android are you running the tests on 
when using the emulator?
What assertion failed message does assertCurrentActivity() give you when it 
fails? Is the issue that it wrongly reports back that another activity is 
shown? Any patterns when looking at the assertion messages?

Original comment by renasr...@gmail.com on 15 Oct 2012 at 8:29

GoogleCodeExporter commented 9 years ago 
I run 4.0.3 normally, though I tested it on 4.1 today and came to the same 
problem. Extending all of my waits out to 10s or more helped, but still has not 
cleared the problem.
The fail message is the string that I hand it, "Activity Wrong" and "expected 
<...> but was <...>". The assertion messages don't seem to have any pattern. 
The assertions tend to fail on at a different location than the previous run.

I did some more testing, and it seems to be a problem with the emulator. I made 
a new emulator of 4.0.3 using the x86 Atom and an HVGA screen ( instead of the 
standard WVGA ) and everything seems to work fine. I will do some more testing, 
but I am 80% confident that this is a problem with the emulator, and not 
Robotium.

Original comment by Phill.Z...@gmail.com on 15 Oct 2012 at 9:31

GoogleCodeExporter commented 9 years ago 
As with most of these testing frameworks, you "simply" need to make your tests 
more robust. Test will always fail randomly if you don't first make sure you 
are ready for testing the assertion ("synchronize"), specially in emulators 
because they are much slower and seem to be *randomly* slow at points (this is 
extremely annoying). For example, if you're loading something, wait until the 
loading indicator has disappeared before checking if the text is correct. Using 
sleep() calls is very haphazard and not ideal, you will probably end up waiting 
much more time than needed in general and still run into many random failures.

Original comment by david...@gmail.com on 15 Nov 2012 at 3:49

GoogleCodeExporter commented 9 years ago 
Will open up again if new information is given.

Original comment by renasr...@gmail.com on 19 Nov 2012 at 6:49

GoogleCodeExporter commented 9 years ago 
Well,

I've got a similar problem and what I've also found is that the more test 
methods are in a test class, the more of them start failing, though no problems 
if running any of the failed methods separately several times in a row.

Original comment by resident...@gmail.com on 13 Jun 2013 at 2:09

GoogleCodeExporter commented 9 years ago 
I'm also facing same issue

Original comment by Sameena....@gmail.com on 16 Jul 2013 at 7:51

GoogleCodeExporter commented 9 years ago 
i m also facing the same issue

Original comment by narendra...@gmail.com on 4 Jun 2014 at 10:56

GoogleCodeExporter commented 9 years ago 
I also have the same problem. When I run the test on device it runs perfectly, 
but when it comes to emulator and jenkins it fails randomly, passes randomly. 
Not knowing the reason, tried all sorts of things none of them worked for me.

Original comment by ravisrav...@gmail.com on 25 Jul 2014 at 1:17

GoogleCodeExporter commented 9 years ago 
Facing the same issue.. the application loads one activity and the test detects 
it as some other activity(any other activity from same application) and the 
problem increases because the behavior is not similar on each test run.. the 
test failure is not linked to any particular activity.

Original comment by phani...@gmail.com on 15 Sep 2014 at 4:15