facebookarchive / xctool

An extension for Apple's xcodebuild that makes it easier to test iOS and macOS apps.
Apache License 2.0
6.91k stars 738 forks source link

Suite timeout not calculated correctly in all cases #761

Closed hushlorentz closed 4 years ago

hushlorentz commented 5 years ago

We've noticed if the test bundle has a structure like suite -> suite -> tests, the overall timeout for the root suite will not count the tests. Instead, it will multiply the testTimeout by the number of child suites. This can happen when the first suite is "All tests".

We fixed this by recursing through the suite structure. Maybe there's a better way?

In otest-shim/otest-shim/otest-shim.m:

static int64_t totalTestCount(id testCollection)
{
  if ([testCollection isKindOfClass:NSClassFromString(@"XCTestCaseSuite")])
  {
    return  [[testCollection tests] count];
  }

  int64_t testCount = 0;

  for (id test in [testCollection tests])
  {
    testCount += totalTestCount(test);
  }

  return testCount;
}
// If running in a suite, time out if we run longer than the combined timeouts of all tests + a fudge factor.
int64_t testCount = isSuite ? totalTestCount(self) : 1;