State not reset between tests when using Test Orchestrator

[patjackson52]

When clearPackageData: true is set and the Test Orchestrator is used, state should not persist between Scenarios. State is not reset between tests when using test orchestrator. Note: Originally I thought clearPackageData: true must be set for state to be reset with Test Orchestrator. Apparently that is the default behavior, and clearPackageData deals with file system (shared prefs, DB, etc)

How to reproduce: A branch with failing tests are here: https://github.com/cucumber/cucumber-android/pull/57 Run tests with ./gradlew runInstrumentationTests and observe failures.

Changes were made to the cukelator example where a static field is set on first run of MainActivity.onCreate(). If the flag is set in subsequent tests, it finishes the activity and fails the test. Commented settings in build.gradle were uncommented to enable Orchestrator and clearPackageData.

As a sanity check I completed the same test on the testing-samples calculator tests. This PR shows code that pass the tests. https://github.com/patjackson52/testing-samples/pull/1

[aslakhellesoy]

Could you please submit your changes as a pull request against this repo? That way we can inspect the diff to see what you have changed.

[patjackson52]

Could you please submit your changes as a pull request against this repo? That way we can inspect the diff to see what you have changed.

Sure thing, I've updated the description with a link.

[lsuski]

@patjackson52 This state in your case is static field. To clear static field the classloader should be killed and created new one. cucumber has nothing to do with static fields and even discourages to use ones. I don't use orchestrator so I'm not expert in it. Are you sure that the same case works with basic AndroidJunitRunner?

[patjackson52]

@lsuski Test Orchestrator will run each test in its own process. All memory is like a fresh launch when using Test Orchestrator. Yes, I have confirmed this works with AndroidJunitRunner in this sample (without cucumber) https://github.com/patjackson52/testing-samples/pull/1 .

I attempted using the AndroidJunitRunner with the cukelator example and no tests were found.

This would be a great feature to have to be able to clear state between scenarios. I'm going to look into the code and see if I can see what is going on. If you, or anyone, is interested in pairing or sharing knowledge please ping me.

[patjackson52]

@lsuski Actually appears cucumber-android does not use test orchestrator at all, at least with the config in my repro example. The test orchestrator process does not run on the test device when using my repro example.

I have been able to clear state between tests however.. Spoon supports this with the `singleInstrumentationCall = false" option. Appears Spoon does not work with Test Orchestrator, and instead has its own method of running isolated instances of the app for each test.

If I'm correct in that Test Orchestrator does not work with cucumber-android, the readme should be updated. It states in the first paragraph that it DOES support orchestrator. A better wording would be "supports sharding via Spoon" Would be great if cucumber-android did support Test Orchestrator, however that may be a large effort. I am going to attempt using Spoon to fit my needs.

If someone confirms the above, I would gladly make a PR for updating documentation about Test Orchestrator, Spoon, and reseting state between tests.

[rhavran]

Hello @patjackson52 ,

Android Test Orchestrator does not work with Spoon, but it works perfectly fine without it, so the claim that it "DOES support Test Orchestrator" is perfectly valid😃. I have added one more case into your PR example and tested it on the raw Android Test Orchestrator and everything works and passes all the cases, so the flag 'clearPackageData' works too, you can try to run with it and without it (use 'gradlew connectedCheck') .

Probably it should be configured differently or fixed by Spoon.

Best regards, Roman.