Mutation tests on stryker based on e2e tests

[bartekleon]

I was thinking, how good and what results could show us using mutation tests on e2e tests. Using mutation tests on unit testing only shows us "how much coverage is good coverage". So basically: if code is tested and "idiot proof". Using mutation tests on E2E/integration on the other hand, should show us how well, we tested entire codebase, and how much code is actually used. (if we were to write really good E2E tests [like a lot of em]), we could probably be able to find some dead paths [it is possible to find them with unit tests too, but its a bit harder]. It is only an idea, but I think its worth thinking about it, its possibilites, extensions etc.

[nicojs]

Thanks for thinking about it, but it is not a good idea, IMHO.

The tests within Stryker are layered as a (testing) pyramid.

       / \
      /   \
     / e2e \
    /_______\
   /   IT    \
  /___________\
 /  Unit tests \
/_______________\

A big number of unit tests, less but still a bunch of integration tests and a hand full of e2e tests (and maybe some manual tests?).

Stryker contains of packages. Each package has

• a lot of unit tests.
• some integration tests, testing integrations with outside tools, like test runners, the file system, typescript compiler, etc.

Last but definitely not least, we have about 25 "e2e" tests (see e2e dir), where we test Stryker in a number of use cases. I like a directory there for each use case (examples: vue-cli-typescript-mocha, vue-cli-javascript-jest, jest-react-enyme-ts, ...)

In the unit tests, you're testing the nitty-gritty. All edge cases and flows should be tested there. The fact that this code isn't hit in integration tests or e2e tests doesn't mean that it is "dead code", rather it means that it is simply not tested in an e2e test. Testing everything with e2e tests is a daunting task and very inefficient. We have 1000's of unit tests, so imagine having 1000's of e2e tests. Not fun at all.

Knowing what to test in what layer is an art form in some way. It isn't always clear at first, but you'll get better at it as time passes.

[Lakitna]

I agree with Nico, the test pyramid is an important model. I've seen applications that used the dreaded test ice cone instead:

\----------------/
 \     E2E      /
  \------------/
   \    IT    /
    \--------/
     \ Unit /
      \    /
       \  /
        \/

Doing this results in a massive amount of test effort and a lot of headaches. Test runtime also skyrockets. Imagine mutation testing a test suite that takes 2 hours to run normally! Mutation testing seems to be more suited to low-level tests because of their speed. I personally use it on my unit tests (about 1000) and the few integration tests (about 5) I have. I don't use it on the E2E tests (about 35).

could probably be able to find some dead paths

This is a great point, but we don't have to rely on mutation coverage for this. Branch coverage or line coverage can reveal this too. That would be a better fit since these kinds of coverage only require a single pass, where mutation coverage requires many. If you're interested in this, try running the E2E tests with nyc https://www.npmjs.com/package/nyc.

We might even be able to get nyc working with a performance benchmark if you're really interested ;)

[bartekleon]

Welp, its just there is possibility we have some code which is simply not actually used but covered in unit tests. Running it against integration / e2e tests should at least help to locate places like these. Actually I have found some dead code recently and I have been preparing PR for it :x

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.