realm / realm-swift

Realm is a mobile database: a replacement for Core Data & SQLite
https://realm.io
Apache License 2.0
16.31k stars 2.14k forks source link

Consider looking into PICT to reduce the number of combinations tested during PRs #4722

Open emanuelez opened 7 years ago

emanuelez commented 7 years ago

PICT is a tool to allow pairwise combinatorial testing (http://www.pairwise.org/index.asp), meaning that, given some axis, it will find the smallest set of combinations that ensures that all pairs are tested.

I gave it a quick try with the cocoa repo and providing this input:

xcode_version: 8.0, 8.1, 8.2
target: osx, docs, ios-static, ios-dynamic, ios-swift, osx-swift, watchos, cocoapods, swiftlint, tvos, osx-encryption, osx-object-server, ios-device-objc, tvos-device
configuration: Debug, Release

it came up with this output:

xcode_version   target  configuration
8.2 ios-dynamic Release
8.1 ios-device-objc Debug
8.0 osx-swift   Release
8.2 tvos-device Debug
8.1 tvos-device Release
8.1 ios-static  Release
8.1 docs    Release
8.0 osx Debug
8.1 cocoapods   Release
8.2 ios-device-objc Release
8.2 osx-swift   Debug
8.2 osx-object-server   Debug
8.2 watchos Debug
8.1 ios-swift   Release
8.0 cocoapods   Debug
8.1 tvos    Debug
8.2 ios-static  Debug
8.0 osx-encryption  Release
8.0 watchos Release
8.0 ios-static  Debug
8.0 swiftlint   Debug
8.0 ios-swift   Debug
8.1 osx Release
8.2 osx Release
8.2 tvos    Release
8.1 swiftlint   Release
8.0 ios-dynamic Debug
8.2 docs    Debug
8.1 watchos Debug
8.0 tvos    Release
8.1 ios-dynamic Debug
8.1 osx-encryption  Debug
8.0 osx-object-server   Release
8.2 swiftlint   Release
8.2 cocoapods   Release
8.2 ios-swift   Release
8.0 tvos-device Release
8.1 osx-object-server   Debug
8.1 osx-swift   Debug
8.0 ios-device-objc Debug
8.2 osx-encryption  Debug
8.0 docs    Debug

This is still slightly longer than the enabled configurations in the current matrix (41 vs 39), but the input file could be improved providing constraints that would allow to further improve the result.

jpsim commented 7 years ago

@emanuelez moving to idea backlog, maybe you could move this entirely to realm/ci though if someone from the automation team will be doing this?