search

muter-mutation-testing / muter

🔎 Automated mutation testing for Swift 🕳️
MIT License
496 stars 39 forks source link

Parallelize testing #64

Open ZevEisenberg opened 5 years ago

ZevEisenberg commented 5 years ago

Muter should run tests in parallel to make the overall test run faster. Maybe we only do this if the testing scheme we are using has the "parallelize tests" option enabled, to make it opt-in. Care should be taken when collecting the results at the end to make sure it is done in a thread-safe way.

SeanROlszewski commented 5 years ago

I love this suggestion, @ZevEisenberg.

I'm thinking there should be a spike to measure how much, if any, parallezing the tests improves performance. I have a concern that because xcodebuild handles multithreading, that there may not be a meaningful improvement, or that there may be a detriment to performance.

I'm having a hard time conceptualizing the performance impacts of a concurrency model for Muter because of how much it depends on other executables which ultimately determine its runtime, and because concurrency is also hard. 😞

A spike of this feature suggestion could be as simple as parallezing the application of mutation operators and the execution of the test suite under test. These changes should be instrumented to prove there is an actual performance improvement.