MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.18k stars 231 forks source link

`bundleIdentifier != nil` runtime crash #545

Open mattrobmattrob opened 1 year ago

mattrobmattrob commented 1 year ago

bundleIdentifier != nil issue when running rules_ios/Bazel-built ios_ui_test that doesn't occur when using the standard xctestrunner.

In this simplistic example, I was getting it to happen always. But in our real project, this crash was happening most of the time (maybe 9/10 cases) but sometimes the app + UI test bundle would launch.

Reproduction

  1. Checkout https://github.com/mattrobmattrob/rules_swift_testbed/tree/mr/bluepill/bundle.identifier.issue. This is a prebuilt "release" on ~master from this repo.
  2. Use the mr/bluepill/bundle.identifier.issue branch
  3. Run bazelisk test //bluepill:UI_Tests_Bluepill.

Runtime Error

{41232} 20221109.230820 [  INFO  ] (BP-1) Test plan started!
{41232} 20221109.230820 [  INFO  ] (BP-1) Test bundle is connected. Capabilities = <XCTCapabilities: 0x600002b573a0>: {
    "ubiquitous test identifiers" = 1;
}
{41232} 20221109.230820 [  INFO  ] (BP-1) _XCT_didBeginExecutingTestPlan
{41232} 20221109.230820 [  INFO  ] (BP-1) All Tests started.
{41232} 20221109.230820 [  INFO  ] (BP-1) Starting TestSuite All tests
{41232} 20221109.230820 [  INFO  ] (BP-1) Starting TestSuite SimpleTest.xctest
{41232} 20221109.230820 [  INFO  ] (BP-1) Starting TestSuite SimpleUITests
{41232} 20221109.230820 [ FAILED ] (BP-1)   0.002188s SimpleUITests/testPasses()
{41232} 20221109.230820 [  ERROR ] (BP-1) HOW DID WE GET AN ERROR THAT WASN'T PARSED? We received an error in a test case that wasn't started or did not parse properly.
Problem line: XCTestOutputBarrier/tests/ios/ui-test/simple.swift:10: error: -[SimpleTest.SimpleUITests testPasses] : Invalid parameter not satisfying: bundleIdentifier != nil (NSInternalInconsistencyException)
{41232} 20221109.230820 [  INFO  ] (BP-1) _XCT_didFinishExecutingTestPlan
{41232} 20221109.230820 [  INFO  ] (BP-1) All Tests Completed.
{41232} 20221109.230820 [  INFO  ] (BP-1) DTXSocketTransport disconnected
{41232} 20221109.230820 [  INFO  ] (BP-1) DTXConnection disconnected.
{41232} 20221109.230821 [  INFO  ] (BP-1) Finished
parikhparth23 commented 1 year ago

@ob @chenxiao0228 can you please have a look?

ob commented 1 year ago

I played with this a bit today (thanks @mattrobmattrob for providing a test case). There are at least two bugs exposed by the test case. One seems easy enough to fix, the other is more puzzling (the bundleIdentifier != nil part).

@nanwng or @chenxiao0228 I can catch you up if you want to take a look.

ob commented 1 year ago

Probably related: https://github.com/bazel-ios/rules_ios/pull/605