Strange behaviours on big projects

[GoogleCodeExporter]

What steps will reproduce the problem?
1.Create test steps with one intent flagged ACTIVITY_CLEAR_TOP or other flags 
that Robotium conception does not take into account.
2.Call solo.finishOpenedActivities()

What is the expected output? What do you see instead?
My test should terminate. Instead, I can get functional exceptions, 
outOfMemoryExceptions due to activities that are unfrozen while they should no 
more exist, hanging InstrumentationTestRunner, ...

What version of the product are you using? On what operating system?
Will crash on any SDK and every version of robotium

Please provide any additional information below.
This is a conception flaw. I think the problem was raised on issue 219, and 
rejected. But the problem does exist.
The whole conception of keeping a hard reference upon the "stack" of activities 
is wrong. Android activity manager is much more complex than a simple stack, 
and stacking activities into an inner object leaks the robotium tests, and 
bypasses the application expected functional workflow. When a hard reference is 
kept upon an activity, you prevent Android from managing activities and garbage 
collecting objects the way it wants to, then the tested application behaves 
strangely : OutOfMemory (because you keep reference upon an activity that has 
reference upon bitmaps that can not be garbagecollected), functional exceptions 
because finishOpenedActivities() calls finish() on activities that should no 
more exist (for example when an intent is started with flag ACTIVITY_CLEAR_TOP, 
or NO_HISTORY), hence implemented activity does not handle this unnatural 
behaviour.
A temporary patch should be to keep a stack of WeakReference<Activity> within 
the activity stack, to prevent robotium from leaking the whole application then 
randomly crashing the instrumentationtestrunner. But I don't get the point of 
keeping the stack at all... Why should this be necessary in a blackbox test ? 

Robotium is a great framework, but this point of conception really is painful. 
This problem keeps crashing our Jenkins continuous integration, holding all 
executors whenever it happens, and forcing us to allocate 256Mo heaps emulators 
to prevent OutOfMemory ! 
If I can help in any way, please let me know.

Original issue reported on code.google.com by alex.gi...@gmail.com on 18 Apr 2012 at 10:18

[GoogleCodeExporter]

What happens if you use finishInactiveActivities() through your test cases. Do 
you still get crashes and OutOfMemoryExceptions?

Original comment by renasr...@gmail.com on 18 Apr 2012 at 11:27

[GoogleCodeExporter]

First, thank you for your responsiveness.

finishInactiveActivities() partially corrects the problem if it is followed by 
getCurrentActivity().finish(). I did not test it through all of my test cases 
(120 minutes of execution, I lack time). But this is only bypassing the 
problem. When you call finish on activities that are not active, you may 
trigger other problems, like infinite loops inside the inner android workflow 
manager (I just had an example a few minutes ago). This solution seems to solve 
my first problem, but may cause other issues for other applications.

The OutOfMemory problem is not solved anyway, because of the inner stack, which 
keeps hard references upon activities. When I study heap size via ddms, it 
appears that layouts, views, business objects and big bitmaps that should be 
freed are never released inside a test.
The point is that in any case, it is a BAD pattern to keep a hard link on a 
sdk-managed object, particularly activities ( 
http://developer.android.com/resources/articles/avoiding-memory-leaks.html ). 
Also, calling finish() on an activity is more than a technical operation, since 
it may have an applicative meaning, trigger other operations, like resuming an 
other activity, calling onActivityResult(), etc... 

Original comment by alex.gi...@gmail.com on 18 Apr 2012 at 1:30

[GoogleCodeExporter]

The issue here is that we can not break backward compatibility 
(getOpenedActivities()). Also this is not only a blackbox test framework, 
people may be interested in getting a hold of the activities that have been 
opened. Another reason for the Activity references to be kept is due to being 
able to finish them at the end. That is needed as having a old Activity still 
open and in focus will lead to complications with the next test case. Many 
times activities are not cleaned properly and therfore manually calling finish 
is a must. 

Using weak references could work. I will send you a version where Robotium uses 
weak references to the Activities instead. Please let me know if it works 
better for you.

Original comment by renasr...@gmail.com on 19 Apr 2012 at 9:00

[GoogleCodeExporter]

Great !
No more memory leak ^^
I created a standard emulator with 25Mo heap, and the used memory is now 
between 55% and 62% maximum. Big objects are garbage collected as soon as huge 
activities are released. (before, I had to use a 256Mo heap, and it was 85% 
full in 2 tests, then the garbage collector had to struggle for survival)
Looks like the functional issue is solved as well : because of weak references, 
the activities that should no more exist are just forgotten ; then the finish() 
that used to trigger strange behaviours is no more called.

I just encountered a (major) side effect issue. It seems that sometimes, the 
"current activity" is not recognized correctly (1 transition on 10, solo is 
late and pretends that the current activity is the previous one). I could not 
step into the source code to find out why, but I guess the jar you sent me was 
a first draft, and you need time to make it stable.

Anyway thank you for your responsiveness, I continue reading the posts, and I 
hope I can help finalizing this major improvement. Don't hesitate if you need 
me to test the next snapshots.

Original comment by alex.gi...@gmail.com on 19 Apr 2012 at 1:46

[GoogleCodeExporter]

Thanks for the great news. I have identified the getCurrentActivity() issue as 
well and I know why it happens. It will be fixed and commited soon. I will send 
you a new jar when all the work has been performed. Thanks for helping out with 
the testing.

Original comment by renasr...@gmail.com on 19 Apr 2012 at 2:05

• Changed state: Accepted

[GoogleCodeExporter]

I have sent you a new jar where all issues should be fixed. Please let me know 
if it works well for you. 

Original comment by renasr...@gmail.com on 26 Apr 2012 at 7:06

[GoogleCodeExporter]

Sorry, I was on holidays the past week.
I'm trying the new jar now, I'll tell you in a few minutes if it works well.

Original comment by alex.gi...@gmail.com on 30 Apr 2012 at 12:34

[GoogleCodeExporter]

Congratulations.
Everything is working perfectly for me. No leak, no getCurrentActivity() issue.
My full testSuite's memory profile is light and clean.

Thank you very much !

Original comment by alex.gi...@gmail.com on 30 Apr 2012 at 1:16

[GoogleCodeExporter]

Thanks for reporting the issue and helping out with the testing. I hope it's ok 
if I send you one last jar before release. 

Original comment by renasr...@gmail.com on 30 Apr 2012 at 1:33

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Of course it's OK !
Thank you for your work again.

Original comment by alex.gi...@gmail.com on 2 May 2012 at 4:44

[GoogleCodeExporter]

Hi Renas

Would you please share the jar of this with me too. Also any ETA on the new 
version will be highly helpful. I have been facing this issue and following 
this thread for a long time now and I am still stuck at OutOfMemory issues. 
Please help me out here.

Original comment by kapilgod...@gmail.com on 29 May 2012 at 10:51

[GoogleCodeExporter]

Hi Renas,

Could you please send me the jar file as well?  Like others I was wondering 
when you would be releasing this version.

Thanks,
Mark

Original comment by markper...@gmail.com on 8 Jun 2012 at 3:10

[GoogleCodeExporter]

Next version will be released early next week. 

Original comment by renasr...@gmail.com on 8 Jun 2012 at 3:12

[GoogleCodeExporter]

This has been fixed in Robotium 3.3. 

Original comment by renasr...@gmail.com on 11 Jun 2012 at 4:11

• Changed state: Verified

[GoogleCodeExporter]

Hi Renas,

the Out of Memory Exception is still happening on my project :(. I am using the 
new Robotium Version 3.3. My tearDown() Method look like this:

public void tearDown() throws Exception {

        try {
            getActivity().finish();
            solo.finishOpenedActivities();

        } catch (Throwable e) {
            e.printStackTrace();
        }
}

I am calling the tearDown method after each testmethod. I have around 100 
automated tests. On every test a user has to login to my app and is performing 
some action.

Do you have an idea what else could go wrong?

Greets,

Daniel

Original comment by dnlkn...@gmail.com on 12 Jun 2012 at 11:38

[GoogleCodeExporter]

What if you do the following?

public void tearDown(){
solo.finishOpenedActivities();
}

Original comment by renasr...@gmail.com on 12 Jun 2012 at 11:41

[GoogleCodeExporter]

If I just do 
public void tearDown(){
   solo.finishOpenedActivities();
}
Just one testmethod is executed. At the end of the test, the testsuite is just 
stopping (not with Out Of Mem Exc.) but it looks like the old activity is still 
open. When I use the tearDown Method I posted before, the tests are running 
fine as long as the OutOfMem Exc. is happening. 

Greets,

Daniel

Original comment by dnlkn...@gmail.com on 12 Jun 2012 at 11:52

[GoogleCodeExporter]

What if you change your setUp() to look like this?

setUp(){
solo = new Solo(getInstrumentation());
getActivity();
}

Original comment by renasr...@gmail.com on 12 Jun 2012 at 11:54

[GoogleCodeExporter]

The same. The test execution is stopping after the first test. If I use 

getActivity().finish();
solo.finishOpenedActivities();

in teardown it is running (with the OOM Exc.)

Original comment by dnlkn...@gmail.com on 12 Jun 2012 at 12:53

[GoogleCodeExporter]

I have emailed you a new jar. Please let me know if you have the same issue. 

Original comment by renasr...@gmail.com on 12 Jun 2012 at 2:02

[GoogleCodeExporter]

Sorry, still getting the out of memory exceptuin...

Original comment by dnlkn...@gmail.com on 12 Jun 2012 at 2:31

[GoogleCodeExporter]

I actually do not know what your issue might be. I dont think its Robotium that 
is causing it. I cant see what in Robotium might be causing this. What does the 
log say? 

Original comment by renasr...@gmail.com on 12 Jun 2012 at 2:35

[GoogleCodeExporter]

That is the only error I receive:

06-12 16:20:42.415: E/AndroidRuntime(25800): Caused by: 
java.lang.OutOfMemoryError: bitmap size exceeds VM budget

I am testing on a real device (Samsung Galaxy S II). The exception occur after 
~20 test executions.

Original comment by dnlkn...@gmail.com on 12 Jun 2012 at 2:37

[GoogleCodeExporter]

Do you get the same issue when running on the emulator? Or other phone?

Original comment by renasr...@gmail.com on 12 Jun 2012 at 2:42

[GoogleCodeExporter]

yes. I tested it with different simulator settings vm size ... and also with 
different devices (Samsung S3, Galaxy Nexus, Motorola Droid and many more) 
always the same. 

I am now trying this: 

protected void tearDown() throws Exception {
   solo.finishInactiveActivities();
   solo.finishOpenedActivities();
}

Original comment by dnlkn...@gmail.com on 12 Jun 2012 at 2:45

[GoogleCodeExporter]

finishOpenedActivities() does the same and more as finishInactiveActivities(). 
If its always at the same point that it happens you can add 
finishInactiveActivities() at that point in order to kill of activities. 
However that should not be needed now when we are using weak references...

Original comment by renasr...@gmail.com on 12 Jun 2012 at 2:49

[GoogleCodeExporter]

When I use finishInactiveActivities() I get the following NPE Exception

06-12 16:52:07.405: W/System.err(29728): java.lang.NullPointerException
06-12 16:52:07.405: W/System.err(29728):    at 
com.jayway.android.robotium.solo.ActivityUtils.finishActivity(ActivityUtils.java
:348)
06-12 16:52:07.405: W/System.err(29728):    at 
com.jayway.android.robotium.solo.ActivityUtils.finishInactiveActivities(Activity
Utils.java:295)
06-12 16:52:07.405: W/System.err(29728):    at 
com.jayway.android.robotium.solo.Solo.finishInactiveActivities(Solo.java:1960)

Original comment by dnlkn...@gmail.com on 12 Jun 2012 at 2:57

[GoogleCodeExporter]

but the test app is not crashing

Original comment by dnlkn...@gmail.com on 12 Jun 2012 at 2:58

[GoogleCodeExporter]

Yes. The exception is catched. It can happen and is according to design. 

Original comment by renasr...@gmail.com on 12 Jun 2012 at 3:01

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi

I am still getting this issue in my build too. Is there a specific way to 
implement the setup method because I tried the one mentioned above and also 
tried using both finishInactiveActivities() and  finishOpenedActivities() in 
teardown method.

Regards
Kapil

Original comment by kapilgod...@gmail.com on 13 Jun 2012 at 10:54

[GoogleCodeExporter]

What's the status of this issue? 

I'm writing a suite case capturing screenshots in port/land for six locales of 
three activities. What are the best practices for achieving this?

At first I wrote a number of ActivityInstrumentationTestCase2 for each locale 
and orientation and made sure that in tearDown() I do 
finishInactiveActivities(). The app crashes with "java.lang.OutOfMemoryError: 
bitmap size exceeds VM budget" and I can see the weak references to the 
activities in the hprof just before the app runs out of memory.

I then tried to put all test cases in one ActivityInstrumentationTestCase2 with 
the hope that this will reduce the weak references by orientations * locales. 
However, that produced the same result.

Any suggestions on how to fix this? Thanks.

Original comment by mobilek...@googlemail.com on 15 Jun 2012 at 10:42

[GoogleCodeExporter]

Please use finishOpenedActivities() in your tearDown() and not 
finishInactiveActivities(). 

Can anyone send me a project and test project where this behaviour can be 
reproduced? I am not able to reproduce it my self even with test suites of 150 
test cases.

Original comment by renasr...@gmail.com on 15 Jun 2012 at 10:47

• Changed state: Accepted

[GoogleCodeExporter]

I wish I could send you my projects however I can't do that because of the IP 
involved. I'll try your suggestion.

Original comment by mobilek...@googlemail.com on 15 Jun 2012 at 11:03

[GoogleCodeExporter]

@Override
protected void tearDown() throws Exception {
   super.tearDown();
   solo.finishOpenedActivities();
}

Once this code is invoked the next test case hangs indefinitely. 

A way to reproduce it is to have a couple of activities and make sure they hold 
on to a large bitmap, byte[], etc.. and write test cases around logic 
revisiting those activities sequentially. 

Can you send me the project you have with the 150 test cases? Thanks.

Original comment by mobilek...@googlemail.com on 15 Jun 2012 at 11:18

[GoogleCodeExporter]

Unfortunately those are 2 test suites that I am not allowed to share. Would it 
be possible for you to create a sample app where the issue you are experiencing 
(where the test case hangs) and send it to me? 

Original comment by renasr...@gmail.com on 15 Jun 2012 at 11:23

[GoogleCodeExporter]

I also have problems with those OutOfMemory crashes. 

I created a sample proect/testproject. The project has two Activities that both 
display very big images. The test project then tries to jump between those 
Activities. When going back to Activity1 the test crashes. Try it yourself!

http://www.file-upload.net/download-4488607/bigimages_test_outofmemory_example.z
ip.html

Original comment by JoaRe...@gmail.com on 29 Jun 2012 at 7:42

[GoogleCodeExporter]

The project also crashes if not under test. Have to figure out another sample 
project. Sorry! 

Original comment by JoaRe...@gmail.com on 29 Jun 2012 at 7:51

[GoogleCodeExporter]

Thanks for helping out with trying to reproduce this!

Original comment by renasr...@gmail.com on 29 Jun 2012 at 1:47

[GoogleCodeExporter]

This issue has finally been fixed in 3.4. Thanks to Daniel Knott for helping 
out with the testing.

Original comment by renasr...@gmail.com on 7 Aug 2012 at 8:54

• Changed state: Verified

[GoogleCodeExporter]

Hi Renas, can you please send me the build which was shared with alex. In all 
my test cases I have to login and then do smething and then logout. But this 
Arraylist keeps on holding references to all the activities which are holding 
references to the bitmap arrays and other java objects. I am also attaching the 
screenshot with the arraylist. Can't send you an expanded one cause of IP 
issues. let me know if any other detail is required.
This is how my tearDown looks like

protected void tearDown() throws Exception {

        //Log out...

        solo.sendKey(Solo.MENU);
        solo.clickOnText("More");
        solo.clickOnText("Logout");
        if (solo.searchButton("Logout")) {
            solo.clickOnButton("Logout");
        }

        solo.waitForActivity(LoginActivity.class.getName(), 1000);

        assertTrue(solo.searchText("Your id"));

        int i = 0;

        while(solo.getAllOpenedActivities()!=null && solo.getAllOpenedActivities().size()>0){

            solo.finishOpenedActivities();
            solo.goBack();                                  

            if(i++>5){
                break;
            }
        }

        solo.finishOpenedActivities();

        solo = null;
        System.gc();
}

Original comment by kukreti....@gmail.com on 14 Aug 2012 at 11:11

Attachments:

• arraylist_with_references_to_activities.bmp

[GoogleCodeExporter]

Please update to 3.4.1 and change your teardown to:

protected void tearDown() throws Exception {

        //Log out...

        solo.sendKey(Solo.MENU);
        solo.clickOnText("More");
        solo.clickOnText("Logout");
        if (solo.searchButton("Logout")) {
            solo.clickOnButton("Logout");
        }

        solo.waitForActivity(LoginActivity.class.getName(), 1000);
        assertTrue(solo.searchText("Your id"));

        solo.finishOpenedActivities();
}

Original comment by renasr...@gmail.com on 14 Aug 2012 at 11:22

[GoogleCodeExporter]

Thanks for youe quick response. I tried using the above mentioned tear down 
method with 3.4.1 but the solo.finishOpenedActivities doesn't finish the login 
activity and the test suite hangs there itself. The method I was using makse it 
move but after running some 45 test cases it runs out of memory. Am I missing 
anything here?

Original comment by kukreti....@gmail.com on 14 Aug 2012 at 12:20

[GoogleCodeExporter]

What you are describing is another issue. I have emailed you about it.

Original comment by renasr...@gmail.com on 14 Aug 2012 at 12:22

[GoogleCodeExporter]

I also have problems with the test suite hangs there itself.How to  resolve the 
problems?

Original comment by qmxiu...@gmail.com on 3 Sep 2012 at 5:27

[GoogleCodeExporter]

I also have this issue on Emulator. It didn't happen on real device.

Project here:
https://code.google.com/p/anymemo/source/browse/#git%2Ftest%2Fsrc%2Forg%2Flibert
y%2Fandroid%2Ffantastischmemo%2Ftest%2Fui

Original comment by mrlhwlib...@gmail.com on 4 Dec 2012 at 9:58