search

net-a-porter-mobile / XCTest-Gherkin

Gherkin syntax for XCTestCase
Apache License 2.0
254 stars 64 forks source link

UI tests crashing on Xcode 11.5+ #167

Closed danspndl closed 10 months ago

danspndl commented 4 years ago

When running UI tests on Xcode 11.5+, instead of reporting test failures, there's an exception that's stopping the whole test run.

The exception seems to be coming from testCase(_:didFailWithDescription:inFile:atLine:) in XCTest-Gherkin. This method is now deprecated

Stack trace:

``` *** Terminating app due to uncaught exception '_XCTestCaseInterruptionException', reason: 'Interrupting test' *** First throw call stack: ( 0 CoreFoundation 0x00007fff23e3de6e __exceptionPreprocess + 350 1 libobjc.A.dylib 0x00007fff512539b2 objc_exception_throw + 48 2 CoreFoundation 0x00007fff23e3dcac +[NSException raise:format:] + 188 3 XCTest 0x000000010f011362 -[XCTestCase(Failures) _recordFailureWithDescription:inFile:atLine:expected:] + 487 4 XCTest 0x000000010efb60ed -[XCTestCase recordFailureWithDescription:inFile:atLine:expected:] + 66 5 XCTest_Gherkin 0x0000000113e3a2d1 $s14XCTest_Gherkin0B5StateC8testCase_22didFailWithDescription6inFile6atLineySo0aE0C_S2SSgSitF + 1313 6 XCTest_Gherkin 0x0000000113e3a5dd $s14XCTest_Gherkin0B5StateC8testCase_22didFailWithDescription6inFile6atLineySo0aE0C_S2SSgSitFTo + 285 7 XCTest 0x000000010efce9a0 -[XCTestObservationCenter _testCaseDidFail:withDescription:inFile:atLine:] + 578 8 XCTest 0x000000010efab10c -[XCTestCaseRun recordFailureWithDescription:inFile:atLine:expected:] + 161 9 XCTest 0x000000010f01123a -[XCTestCase(Failures) _recordFailureWithDescription:inFile:atLine:expected:] + 191 10 XCTest 0x000000010efb60ed -[XCTestCase recordFailureWithDescription:inFile:atLine:expected:] + 66 11 XCTest 0x000000010f01093b -[XCTestCase(Failures) _dequeueFailures] + 859 12 XCTest 0x000000010f010dfb -[XCTestCase(Failures) _enqueueFailureWithDescription:inFile:atLine:expected:interruptTest:breakWhenDequeued:] + 803 13 XCTest 0x000000010efc504c _XCTFailureHandler + 516 14 XCTest 0x000000010efc55e7 _XCTPreformattedFailureHandler + 155 15 libXCTestSwiftSupport.dylib 0x000000011e81d56b $s6XCTest13XCTAssertTrue__4file4lineySbyKXK_SSyXKs12StaticStringVSutF + 1179 16 App-UITests-Runner 0x0000000111227547 $s13appUITest28ScreenStepsC06defineF0yyFySicfU4_ + 199 17 App-UITests-Runner 0x000000011122767f $sSiIegy_SiIegn_TR + 15 18 XCTest_Gherkin 0x0000000113e30efb $s14XCTest_Gherkin11StepDefinerC4step_4file4line1fySS_SSSiyxctAA26MatchedStringRepresentableRzlFyAA0C7MatchesVySSGcfU_ + 1003 19 XCTest_Gherkin 0x0000000113e30fcc $s14XCTest_Gherkin11StepDefinerC4step_4file4line1fySS_SSSiyxctAA26MatchedStringRepresentableRzlFyAA0C7MatchesVySSGcfU_TA + 60 20 XCTest_Gherkin 0x0000000113e42914 $sSo10XCTestCaseC0A8_GherkinE11performStep_7keyword4file4lineySS_S2SSitFySo11XCTActivity_pXEfU0_ + 900 21 XCTest_Gherkin 0x0000000113e42b43 $sSo10XCTestCaseC0A8_GherkinE11performStep_7keyword4file4lineySS_S2SSitFySo11XCTActivity_pXEfU0_TA + 51 22 XCTest_Gherkin 0x0000000113e42b5f $sSo11XCTActivity_ps5Error_pIggzo_SoAA_pytsAB_pIeggrzo_TR + 15 23 XCTest_Gherkin 0x0000000113e42bb4 $sSo11XCTActivity_ps5Error_pIggzo_SoAA_pytsAB_pIeggrzo_TRTA + 20 24 libXCTestSwiftSupport.dylib 0x000000011e819e48 $sSo10XCTContextC6XCTestE11runActivity5named5blockxSS_xSo11XCTActivity_pKXEtKlFZxyKXEfU_ + 152 25 libXCTestSwiftSupport.dylib 0x000000011e819ea1 $sSo10XCTContextC6XCTestE11runActivity5named5blockxSS_xSo11XCTActivity_pKXEtKlFZxyKXEfU_TA + 33 26 libswiftObjectiveC.dylib 0x000000011f303c6e $s10ObjectiveC15autoreleasepool8invokingxxyKXE_tKlF + 46 27 libXCTestSwiftSupport.dylib 0x000000011e819d45 $sSo10XCTContextC6XCTestE11runActivity5named5blockxSS_xSo11XCTActivity_pKXEtKlFZ + 197 28 XCTest_Gherkin 0x0000000113e41f26 $sSo10XCTestCaseC0A8_GherkinE11performStep_7keyword4file4lineySS_S2SSitF + 6390 29 XCTest_Gherkin 0x0000000113e42c82 $sSo10XCTestCaseC0A8_GherkinE11performStep_7keyword4file4lineySS_S2SSitFTo + 194 30 XCTest_Gherkin 0x0000000113e3dfde $sSo10XCTestCaseC0A8_GherkinE4Then_4file4lineySS_SSSitF + 254 31 XCTest_Gherkin 0x0000000113e3e0aa $sSo10XCTestCaseC0A8_GherkinE4Then_4file4lineySS_SSSitFTo + 138 32 App-UITests-Runner 0x0000000111639002 $s13appUITest23AppFeatureC66test_caseyyF + 2722 33 App-UITests-Runner 0x000000011163910b $s13appUITest23AppFeatureC66test_caseyyFTo + 43 34 CoreFoundation 0x00007fff23e44dec __invoking___ + 140 35 CoreFoundation 0x00007fff23e41fd1 -[NSInvocation invoke] + 321 36 XCTest 0x000000010efb6fa7 __24-[XCTestCase invokeTest]_block_invoke_2 + 52 37 XCTest 0x000000010efb6f53 __24-[XCTestCase invokeTest]_block_invoke.206 + 320 38 XCTest 0x000000010f011d32 +[XCTestCase(Failures) performFailableBlock:testCase:testCaseRun:shouldInterruptTest:] + 69 39 XCTest 0x000000010f011c44 -[XCTestCase(Failures) _performTurningExceptionsIntoFailuresInterruptAfterHandling:block:] + 115 40 XCTest 0x000000010efb6966 -[XCTestCase invokeTest] + 1183 41 XCTest 0x000000010efb8299 __26-[XCTestCase performTest:]_block_invoke_2 + 43 42 XCTest 0x000000010f011d32 +[XCTestCase(Failures) performFailableBlock:testCase:testCaseRun:shouldInterruptTest:] + 69 43 XCTest 0x000000010f011c44 -[XCTestCase(Failures) _performTurningExceptionsIntoFailuresInterruptAfterHandling:block:] + 115 44 XCTest 0x000000010efb81d0 __26-[XCTestCase performTest:]_block_invoke.359 + 86 45 XCTest 0x000000010f02497d +[XCTContext runInContextForTestCase:block:] + 211 46 XCTest 0x000000010efb7a84 -[XCTestCase performTest:] + 566 47 XCTest 0x000000010effe2fe -[XCTest runTest] + 57 48 XCTest 0x000000010efb1cc0 __27-[XCTestSuite performTest:]_block_invoke + 354 49 XCTest 0x000000010efb1412 __59-[XCTestSuite _performProtectedSectionForTest:testSection:]_block_invoke + 24 50 XCTest 0x000000010f02497d +[XCTContext runInContextForTestCase:block:] + 211 51 XCTest 0x000000010efb13c9 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 148 52 XCTest 0x000000010efb172e -[XCTestSuite performTest:] + 348 53 XCTest 0x000000010effe2fe -[XCTest runTest] + 57 54 XCTest 0x000000010efb1cc0 __27-[XCTestSuite performTest:]_block_invoke + 354 55 XCTest 0x000000010efb1412 __59-[XCTestSuite _performProtectedSectionForTest:testSection:]_block_invoke + 24 56 XCTest 0x000000010f02497d +[XCTContext runInContextForTestCase:block:] + 211 57 XCTest 0x000000010efb13c9 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 148 58 XCTest 0x000000010efb172e -[XCTestSuite performTest:] + 348 59 XCTest 0x000000010effe2fe -[XCTest runTest] + 57 60 XCTest 0x000000010efb1cc0 __27-[XCTestSuite performTest:]_block_invoke + 354 61 XCTest 0x000000010efb1412 __59-[XCTestSuite _performProtectedSectionForTest:testSection:]_block_invoke + 24 62 XCTest 0x000000010f02497d +[XCTContext runInContextForTestCase:block:] + 211 63 XCTest 0x000000010efb13c9 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 148 64 XCTest 0x000000010efb172e -[XCTestSuite performTest:] + 348 65 XCTest 0x000000010effe2fe -[XCTest runTest] + 57 66 XCTest 0x000000010f033e84 __44-[XCTTestRunSession runTestsAndReturnError:]_block_invoke + 171 67 XCTest 0x000000010f033f71 __44-[XCTTestRunSession runTestsAndReturnError:]_block_invoke.100 + 96 68 XCTest 0x000000010efcc6b6 -[XCTestObservationCenter _observeTestExecutionForBlock:] + 682 69 XCTest 0x000000010f033c0f -[XCTTestRunSession runTestsAndReturnError:] + 615 70 XCTest 0x000000010ef956b4 -[XCTestDriver runTestsAndReturnError:] + 456 71 XCTest 0x000000010f0205bc _XCTestMain + 2496 72 App-UITests-Runner 0x000000010eda2e1a -[_XCTRunnerAppDelegate applicationWillResignActive:] + 0 73 App-UITests-Runner 0x000000010eda2d18 _XCTRunnerRunTests + 0 74 CoreFoundation 0x00007fff23da1abc __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 12 75 CoreFoundation 0x00007fff23da11b3 __CFRunLoopDoBlocks + 195 76 CoreFoundation 0x00007fff23d9bfa3 __CFRunLoopRun + 995 77 CoreFoundation 0x00007fff23d9b8a4 CFRunLoopRunSpecific + 404 78 GraphicsServices 0x00007fff38c39bbe GSEventRunModal + 139 79 UIKitCore 0x00007fff49325968 UIApplicationMain + 1605 80 App-UITests-Runner 0x000000010eda2fd5 main + 183 81 libdyld.dylib 0x00007fff520ce1fd start + 1 ) libc++abi.dylib: terminating with uncaught exception of type _XCTestCaseInterruptionException ```
danspndl commented 4 years ago

The workaround I found for Xcode 11.6 is to remove continueAfterFailure = false. This issue has been resolved in the Xcode 12 betas.

deanWombourne commented 4 years ago

Annoyingly I can't replace it yet because we support the current and current-1 versions of Xcode - the replacement method isn't available in Xcode 11 so can't replace it yet.

Sigh.

Any suggestions welcome - though it's good to know you found a workaround :)