Monkey never stops if app crashes when testing with xcodebuild

[andris-zalitis]

I think SwiftMonkey is very powerful way to find crashes in an app 👍. Ideally I'd like to be able to run it from CLI. That would make it possible to create scripts that run SwiftMonkey non-stop, gather crash reports, screenshots etc.

However, currently if a test is being launched via xcodebuild test SwiftMonkey continues its frenzy even after the app has crashed (after some time, it usually crashes some system apps too 😀). xcodebuild also continues running after the app has crashed.

However if I create UI Test that "manually" executes the same button press that results in a crash, then xcodebuild reports failed test and stops.

The reason why Monkey keeps running is probably related to the fact that XCUIApplication().state returns .runningForeground even after the app has crashed (I checked by adding a line in func monkeyAround()). It seems that somehow Monkey is keeping it alive?

I'm experiencing this behaviour on Xcode 10, didn't try on Xcode 9 yet.

[andris-zalitis]

It seems that somehow Monkey is keeping it alive?

Did a little more of investigation. Looks like it's not that Monkey is keeping the app alive, but since we crashed the app via input event instead of XCUIElement event, crash is not reported to XCUIApplication.

Steps to reproduce:

• Create sample project with button that tries to unwrap IUO that's nil.
• Create UI Test like this:

  func testExample() {
      repeat {
          NSLog("Still running")
      } while (true)
  }

• Run the test with xcodebuild
• Click on the button using your mouse
• Notice how xcodebuild does not stop running even after you crash the app by clicking on the button.

If you'd replace test code with something like XCUIApplication().buttons["Button"].tap() you'd see xcodebuild stopping.

So basically, it seems that in order to get this working we'd need to do something like is mentioned here #2.

[andris-zalitis]

Based on my last comment, I got an idea of how to make the test fail when app has stopped. We can do it if we add something like:

if #available(iOS 9.0, *) {
    _ = XCUIApplication().descendants(matching: .any).element(boundBy: 0).frame
}

in func monkeyAround().

Then after app has crashed the test will fail with:

 Find the Any
     Snapshot accessibility hierarchy for app with pid 9217
     Find the Any (retry 1)
 Assertion Failure: <unknown>:0: Error getting main window -25204
 Tear Down

However I noticed a problem with this approach - no crash report is generated in DerivedData/.../Logs/Test/*.xcresult/ 😞. All we have in this case is a lousy StandardOutputAndStandardError-<bundle-id>.txt that only has the crash error message like Fatal error: Unexpectedly found nil while unwrapping an Optional value without stack trace or any other details.

[wojciechczerski]

@andris-zalitis Thanks for reporting this issue! I'm on it! Will keep you informed.

[wojciechczerski]

Hey @andris-zalitis! I did some tests on my own and I couldn't find any workaround for solving this issue. I think your suggestion

if #available(iOS 9.0, *) {
    _ = XCUIApplication().descendants(matching: .any).element(boundBy: 0).frame
}

... could be the best thing we're having right now. I tried "breaking" the infinite loops SwiftMonkey is using in monkeyAround() by using OperationQueue - didn't work. I was also searching whether it's possible to attach a debugger when testing from command line in hope that it could stop Monkey after a crash (just like Xcode does). No luck here as well.

Let's keep this issue open, perhaps someone might have an idea how it can be solved.

[jihun-im]

I am also trying to run SwiftMonkey from CLI. What I did is

However, SwiftMonkey keep testing even though crash occurred and test application was turned off. Is there something wrong in my code?

What I want to do is making python script to automatically record video, capture log(debug-able) and start SwiftMonkey test.

[wojciechczerski]

Closing the issue due to inactivity. Please reopen it if you think it should be addressed.