Currently, breaking the setup flow for Nimble is easy because of the many supported installation methods in various configurations:
Platforms - Supporting latest and latest - 1 when possible; Assumes the same version of Xcode can be used. Simulator-only is acceptable.
iOS (7, 8, 9) - iOS 7 is incidentally supported in simulator only by setting min version target
macOS (10.9, 10.10, 10.11) - Currently we can only cover one due to CI limitations
tvOS (1)
Linux
macOS without Objective-C Runtime (SPM on mac)
watchOS (Unverified)
Subject Target - The binary target that's being tested. Static libraries and CLI is currently unsupported. Build-processes differ between both targets that may break specific configurations.
None (only tests code in the target target)
Application Bundle
Dynamic Framework - Assume restricted extensions API environment
Test Target - The target that Nimble assertions reside in. This target links directly to Nimble. CLI may be a future target.
App Bundle - Since Nimble supports linking without XCTest
Unit Test Bundle
UI Test Bundle - only applies to app bundles?
Installation Method - How end-users install Nimble into their projects
Embed as subproject (submodule)
Cocoapods
Carthage
Swift Package Manager
Configurations - How end-users / consumers configure Nimble
Linking with/without XCTest
SwiftLint is/isn't installed
This is a combinatorial explosion that's difficult to ensure we don't break anything. Not all combinations are possible (UI Test Bundle w/o XCTest is impossible), but there's still a lot of possible configurations. And it will only get worse as time progresses.
A good first-step is to have an acceptance test suite that pulls in the latest commit of Nimble and runs sample projects in each of those configurations. I definitely don't expect we'll get this coverage with one large PR, but it doesn't hurt to build this over time.
To be clear, these sample projects should simply check that Nimble can be linked + run with those sample projects. There shouldn't be any significant assertions other than running some Nimble code to assert that Nimble successfully links and runs in the configuration.
Question for @Quick/contributors: should this be inside the Nimble project itself or as a separate project under the Quick org? A separate project would require triggering another build as needed (which is a bit more infrastructure work to setup), although I'm leaning towards that.
Some common configurations we should support (in order of importance):
[x] iOS 9; App bundle; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
[x] iOS 9; App bundle; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
[ ] iOS 9; Framework; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
[ ] iOS 9; Framework; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
[x] iOS 9; App bundle; UI Test Bundle; Cocoapods; with XCTest; with SwiftLint
[x] iOS 9; App bundle; UI Test Bundle; Carthage; with XCTest; with SwiftLint
[x] macOS (10.11); App bundle; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
[x] macOS (10.11); App bundle; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
[ ] macOS (10.11); Framework; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
[ ] macOS (10.11); Framework; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
[x] macOS (10.11); App bundle; UI Test Bundle; Cocoapods; with XCTest; with SwiftLint
[x] macOS (10.11); App bundle; UI Test Bundle; Carthage; with XCTest; with SwiftLint
[ ] macOS (10.11); SwiftPM; with XCTest
[ ] watchOS 2; App bundle; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
[ ] watchOS 2; App bundle; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
[x] tvOS; App bundle; Unit Test Bundle; Cocoapods; with XCTest; with SwiftLint
[x] tvOS; App bundle; Unit Test Bundle; Carthage; with XCTest; with SwiftLint
[ ] tvOS; App bundle; UI Test Bundle; Cocoapods; with XCTest; with SwiftLint
[ ] tvOS; App bundle; UI Test Bundle; Carthage; with XCTest; with SwiftLint
[ ] Linux; SwiftPM; with XCTest
[ ] iOS 9; App bundle; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
[ ] iOS 9; App bundle; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
[ ] iOS 9; Framework; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
[ ] iOS 9; Framework; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
[ ] iOS 9; App bundle; UI Test Bundle; Cocoapods; with XCTest; without SwiftLint
[ ] iOS 9; App bundle; UI Test Bundle; Carthage; with XCTest; without SwiftLint
[ ] macOS (10.11); App bundle; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
[ ] macOS (10.11); App bundle; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
[ ] macOS (10.11); Framework; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
[ ] macOS (10.11); Framework; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
[ ] macOS (10.11); App bundle; UI Test Bundle; Cocoapods; with XCTest; without SwiftLint
[ ] macOS (10.11); App bundle; UI Test Bundle; Carthage; with XCTest; without SwiftLint
[ ] watchOS 2; App bundle; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
[ ] watchOS 2; App bundle; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
[ ] tvOS; App bundle; Unit Test Bundle; Cocoapods; with XCTest; without SwiftLint
[ ] tvOS; App bundle; Unit Test Bundle; Carthage; with XCTest; without SwiftLint
[ ] tvOS; App bundle; UI Test Bundle; Cocoapods; with XCTest; without SwiftLint
[ ] tvOS; App bundle; UI Test Bundle; Carthage; with XCTest; without SwiftLint
jeffh
commented
9 years ago
Currently, breaking the setup flow for Nimble is easy because of the many supported installation methods in various configurations:
Platforms - Supporting
latestandlatest - 1when possible; Assumes the same version of Xcode can be used. Simulator-only is acceptable.Subject Target - The binary target that's being tested. Static libraries and CLI is currently unsupported. Build-processes differ between both targets that may break specific configurations.
Test Target - The target that Nimble assertions reside in. This target links directly to Nimble. CLI may be a future target.
Installation Method - How end-users install Nimble into their projects
Configurations - How end-users / consumers configure Nimble
This is a combinatorial explosion that's difficult to ensure we don't break anything. Not all combinations are possible (UI Test Bundle w/o XCTest is impossible), but there's still a lot of possible configurations. And it will only get worse as time progresses.
A good first-step is to have an acceptance test suite that pulls in the latest commit of Nimble and runs sample projects in each of those configurations. I definitely don't expect we'll get this coverage with one large PR, but it doesn't hurt to build this over time.
To be clear, these sample projects should simply check that Nimble can be linked + run with those sample projects. There shouldn't be any significant assertions other than running some Nimble code to assert that Nimble successfully links and runs in the configuration.
Question for @Quick/contributors: should this be inside the Nimble project itself or as a separate project under the Quick org? A separate project would require triggering another build as needed (which is a bit more infrastructure work to setup), although I'm leaning towards that.
Some common configurations we should support (in order of importance):