search

JohnSundell / Marathon

[DEPRECATED] Marathon makes it easy to write, run and manage your Swift scripts 🏃
MIT License
1.86k stars 78 forks source link

Fix CI flakiness #98

Open JohnSundell opened 7 years ago

JohnSundell commented 7 years ago

Our CI can sometimes exhibit some flaky behavior in that it'll fail for no good reason, and a retrigger solves the problem. Let's investigate what's going wrong (maybe it's a caching thing, or some state that isn't properly cleaned up, or some race condition?) and fix it!

Interestingly, the flakiness is different on BuddyBuild vs Travis.

Travis

Here's what a failed build log usually looks like on Travis:

Test Suite 'All tests' started at 08:31:26.018
Test Suite 'debug.xctest' started at 08:31:26.042
Test Suite 'MarathonTests' started at 08:31:26.042
Test Case 'MarathonTests.testInvalidCommandThrows' started at 08:31:26.042
Test Case 'MarathonTests.testInvalidCommandThrows' passed (0.002 seconds)
Test Case 'MarathonTests.testAddingAndRemovingRemotePackage' started at 08:31:26.044
Test Case 'MarathonTests.testAddingAndRemovingRemotePackage' passed (1.959 seconds)
Test Case 'MarathonTests.testAddingAndRemovingLocalPackage' started at 08:31:28.004
Test Case 'MarathonTests.testAddingAndRemovingLocalPackage' passed (0.729 seconds)
Test Case 'MarathonTests.testRemovingAllPackages' started at 08:31:28.733
Test Case 'MarathonTests.testRemovingAllPackages' passed (9.061 seconds)
Test Case 'MarathonTests.testAddingLocalPackage' started at 08:31:37.794
Test Case 'MarathonTests.testAddingLocalPackage' passed (0.54 seconds)
Test Case 'MarathonTests.testAddingLocalPackageWithDependency' started at 08:31:38.334
Test Case 'MarathonTests.testAddingLocalPackageWithDependency' passed (1.286 seconds)
Test Case 'MarathonTests.testAddingLocalPackageWithUnsortedVersionsContainingLetters' started at 08:31:39.620
Test Case 'MarathonTests.testAddingLocalPackageWithUnsortedVersionsContainingLetters' passed (0.914 seconds)
Test Case 'MarathonTests.testAddingAlreadyAddedPackageThrows' started at 08:31:40.534
Test Case 'MarathonTests.testAddingAlreadyAddedPackageThrows' passed (5.021 seconds)
Test Case 'MarathonTests.testTreatingNestedDependenciesAsAdded' started at 08:31:45.555
Test Case 'MarathonTests.testTreatingNestedDependenciesAsAdded' passed (2.605 seconds)
Test Case 'MarathonTests.testRunningScriptWithoutPathThrows' started at 08:31:48.160
Test Case 'MarathonTests.testRunningScriptWithoutPathThrows' passed (0.004 seconds)
Test Case 'MarathonTests.testRunningScript' started at 08:31:48.164
Test Case 'MarathonTests.testRunningScript' passed (3.542 seconds)
Test Case 'MarathonTests.testRunningScriptWithNewDependency' started at 08:31:51.705

travis_time:end:0d599c36:start=1496651472166114307,finish=1496651515992986208,duration=43826871901
The command "swift test" exited with 1.

Done. Your build exited with 1.

As you can see above, it just exits with code 1, no additional debug information seems to be found 😢 It also seems like it stops on different test cases every time this happens, so don't think it's a problem with a single test case. Perhaps a timeout issue?

BuddyBuild

On BuddyBuild things are a lot more consistent, and should be easier to fix. It always fails on this test case:

Test Case '-[MarathonTests.MarathonTests testAddingAlreadyAddedPackageThrows]' started.

fatal error: unexpectedly found nil while unwrapping an Optional value

Current stack trace:
0    libswiftCore.dylib                 0x00000001104ba130 swift_reportError + 129
1    libswiftCore.dylib                 0x00000001104d6b50 _swift_stdlib_reportFatalError + 60
2    libswiftCore.dylib                 0x00000001102c7250 specialized specialized StaticString.withUTF8Buffer<A> ((UnsafeBufferPointer<UInt8>) -> A) -> A + 342
3    libswiftCore.dylib                 0x0000000110441e90 partial apply for (_fatalErrorMessage(StaticString, StaticString, file : StaticString, line : UInt, flags : UInt32) -> Never).(closure #2) + 109
4    libswiftCore.dylib                 0x00000001102c7250 specialized specialized StaticString.withUTF8Buffer<A> ((UnsafeBufferPointer<UInt8>) -> A) -> A + 342
5    libswiftCore.dylib                 0x00000001103f49a0 specialized _fatalErrorMessage(StaticString, StaticString, file : StaticString, line : UInt, flags : UInt32) -> Never + 96
6    MarathonPackageTests               0x00000001100bc200 MarathonTests.tearDown() -> () + 108
7    MarathonPackageTests               0x00000001100bc340 @objc MarathonTests.tearDown() -> () + 34
8    XCTest                             0x000000010dc45657 __24-[XCTestCase invokeTest]_block_invoke_2 + 1492
9    XCTest                             0x000000010dc83b4c -[XCTestContext performInScope:] + 190
10   XCTest                             0x000000010dc45546 -[XCTestCase invokeTest] + 254
11   XCTest                             0x000000010dc45e7e -[XCTestCase performTest:] + 565
12   XCTest                             0x000000010dc42efa __27-[XCTestSuite performTest:]_block_invoke + 300
13   XCTest                             0x000000010dc42bf9 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 29
14   XCTest                             0x000000010dc42d26 -[XCTestSuite performTest:] + 214
15   XCTest                             0x000000010dc42efa __27-[XCTestSuite performTest:]_block_invoke + 300
16   XCTest                             0x000000010dc42bf9 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 29
17   XCTest                             0x000000010dc42d26 -[XCTestSuite performTest:] + 214
18   XCTest                             0x000000010dc42efa __27-[XCTestSuite performTest:]_block_invoke + 300
19   XCTest                             0x000000010dc42bf9 -[XCTestSuite _performProtectedSectionForTest:testSection:] + 29
20   XCTest                             0x000000010dc42d26 -[XCTestSuite performTest:] + 214
21   XCTest                             0x000000010dc96b24 __44-[XCTTestRunSession runTestsAndReturnError:]_block_invoke + 40
22   XCTest                             0x000000010dc5950c -[XCTestObservationCenter _observeTestExecutionForBlock:] + 587
23   XCTest                             0x000000010dc968d1 -[XCTTestRunSession runTestsAndReturnError:] + 281
24   XCTest                             0x000000010dc2e9ac -[XCTestDriver runTestsAndReturnError:] + 254
25   XCTest                             0x000000010dc85fb8 _XCTestMain + 773
26   xctest                             0x000000010dc1c48d <unavailable> + 5261
27   libdyld.dylib                      0x00007fffea83c234 start + 1

Rebuilding without cache seems to fix the problem. Perhaps some cleanup not being made?

cojoj commented 7 years ago

Most of the time when I was getting strange errors in different CIs it was regarding timeouts. We're using free plans, so the bandwidth etc. is relatively slow. Marathon test suite is heavy, so I wouldn't be surprised if it will turn out to be timeout errors.

Regarding buddybuild error. Maybe extracting try? folder.subfolder(named: "Packages").file(named: "Files").read() to a let can will help?

cojoj commented 7 years ago

Feels like Swift 3.2 introduced new problem. Take a look at this integration . I've faced this issue locally as well when I was running tests. It's stuck on testAddingAndRemovingRemotePackage or testAddingLocalPackage.

JohnSundell commented 7 years ago

@cojoj Fixed that issue here: https://github.com/JohnSundell/Marathon/pull/104