search

getsentry / sentry-cocoa

The official Sentry SDK for iOS, tvOS, macOS, watchOS.
https://sentry.io/for/cocoa/
MIT License
808 stars 319 forks source link

Add mutation testing #2096

Closed armcknight closed 1 year ago

armcknight commented 2 years ago

Description

I noticed a change I made in a test fixture did not cause any tests to fail. That got me thinking about the quality of our tests, and mutation testing can help with this. See https://github.com/muter-mutation-testing/muter

Probably best to run on a cron schedule, as it might take some time to complete.

Bonus points if we can get it to also mutate the tests, instead of just the source code under test.

philipphofmann commented 2 years ago

Let's do a spike by taking a look at the https://github.com/muter-mutation-testing/muter and see how valuable the library could be.

philipphofmann commented 2 years ago

It is worth mentioning that the test mutation library must allow ignoring certain code regions. We won't be able to fix everything in one PR. We also need a strategy to ensure new code is not ignored by the test mutation library.

github-actions[bot] commented 2 years ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

philipphofmann commented 1 year ago

@armcknight, while this is a great idea, I will close this as we currently don't have any plans on adding this, and I'd like to keep our backlog small. We can always reopen the issue or add mutation testing if it makes sense.