ChimeHQ / UITestingPlus

Utilities for working with XCUI testing
BSD 3-Clause "New" or "Revised" License
38 stars 2 forks source link

Add various utilities for performing actions "soon" (default 3 seconds). #5

Closed chrisvasselli closed 1 year ago

chrisvasselli commented 1 year ago

Given the sometimes flaky nature of the "wait for app to idle" mechanism, using these functions in place of their standard counterparts is a good way to reduce flaky test failures.

In my personal usage, I almost always use these soon variants in place of their non-soon versions, especially existsSoon and tapSoon. I have found that it reduces flakiness a lot, and avoids a lot of headaches at the yearly iOS release when Apple inevitably subtly changes the behavior of UITesting. When tests are passing, they don't add any extra time overhead, so there's not much of a downside to using them extensively.

Places I don't use them are when I have a series of asserts in a row and I know the state isn't changing between them. Or when I actually care about the timing (although I avoid this as much as possible, since relying on timing usually leads to flakiness).

chrisvasselli commented 1 year ago

I don't use XCTWaiter because to create an expectation that uses a block, I need to wrap it in an NSPredicate, which then requires that the closure be @escapable. This isn't a huge problem, but it requires explicitly adding self. to any referenced member variables, and that means it's no longer a simple drop-in replacement for XCTAssert, etc., and adds a little unnecessary verbosity.

XCTWaiter seems to basically do the same thing I'm doing, querying the app periodically (it looks like every 1.5 seconds) to look for changes. So I don't think there's much difference with my approach.

mattmassicotte commented 1 year ago

This is a really interesting approach! I think the expectation stuff would also work nicely with HittableElementExpectation, which just wraps up the predicate.

The one issue I see is most of these methods heavily lean on try?, and discard thrown errors. I haven't used them, so perhaps this is intentional. But, I would think that makes it a lot harder to understand why things have failed, does it not?

chrisvasselli commented 1 year ago

@mattmassicotte you're totally right, it is eating those errors when it should be sending them to the underlying XCTAssert function. I'll open another PR.