search

kif-framework / KIF

Keep It Functional - An iOS Functional Testing Framework
Other
6.21k stars 914 forks source link

First test on iPad prevents tapping #1225

Closed billburgess closed 3 years ago

billburgess commented 3 years ago

For some reason, the first test in my suite always fails when run on an iPad. Doesn't seem to matter the OS version. I'm not sure what it could be. I've tried commenting out tests and the failure always trickles down stream to the first test that runs. Screenshots show my app in an appropriate spot, but some items are not tappable. My suspicion is that it is tapping too early and it gets missed and then the next item isn't ready.

My only fix has been to program a hard waitForTimeInterval: where I know the first test fails. This is not sustainable long term for our suite. I'm curious if anyone has any thoughts or theories or fixes as to why the first test on an iPad always takes longer to get set up and register for taps.

This issue is more than likely only related to my code base and I can't share anything specific, but was hoping there was some explanation or someone else out there that has had similar issues. To avoid breaking any of my actual tests, I have a dummy test that runs first that I can experiment on to see what works and what doesn't. So far, the only thing that works is either to have a test that can't fail be the first, or to hardcode a wait in that test to let it "catch up", then continue like other tests. Any help would be greatly appreciated.

kenji21 commented 3 years ago

does the issue is reproducible when using random test execution order ? see https://useyourloaf.com/blog/xcode-10-random-and-parallel-tests/

if sometimes it's green, then you have side effect in a previous test maybe you can run only test of a single class to "speed up" the issue reproduction

billburgess commented 3 years ago

The issue is reproducible 100% of the time on our CI server where most of the tests run. We use VM's that run on slightly older hardware. 2015 MBP's. I can't quite reproduce it on my 2018 MBP or when running the same VM locally. It feels like the first test always starts up a little slower.

We have randomization and parallel builds off to ensure any failures can be reproduced accurately enough. We do a pretty good job I think of cleaning up between tests to ensure there isn't a pollution issue between tests, but I can't guarantee that there isn't.

I enabled randomization and the first test did not fail this time. I can't explain why at this time. The first test follows the same startup path as my normally failing test. The randomization did cause a lot of other failures downstream that don't normally come out. My assumption as this is not likely a KIF issue, but I can't for the life of me figure out why. I'm hoping someone has some advice or a way to use KIF to figure out what is going on.

billburgess commented 3 years ago

I was able to mitigate this issue today by pre-starting the simulator. It seems by starting the sim separate from the test command the first test issue seems to resolve itself.