Add Infection Mutation Testing as Library

larsroettig commented 5 years ago

Description

As Developer, I would use Mutation Testing for my Magento Implementations.

Expected behavior

Injection is integrated https://infection.github.io/guide/installation.html

Benefits

It is a powerful approach to attain high coverage of the source program.
This testing is capable comprehensively testing the mutant program.
Mutation testing brings a good level of error detection to the software developer.
This method uncovers ambiguities in the source code, and has the capacity to detect all the faults in the program.
Customers are benefited from this testing by getting most reliable and stable system.

Additional information

paliarush commented 5 years ago

Makes sense to try it out to see how well it works for the Magento codebase. In scope of prototype we should evaluate:

For which kinds of tests mutation testing makes sense (integration, MFTF, web API, etc)
Tests execution time implications
Considering the size of Magento codebase, we might only need to "infest" select modules while running tests for these modules. We need to think about other possible optimizations
Is it possible to get repeatable results? How do we verify that the fixed issue?
Are we going to integrate it into PR delivery process or run once in a while manually?

theofidry commented 5 years ago

Given the size of the Magento codebase, I would really target unit tests since otherwise the waiting time may be too long. Using it incrementally, i.e. just on the files changed, would also make it faster.

Is it possible to get repeatable results? How do we verify that the fixed issue?

If your tests are deterministic yes it is: mutants are generated and applied in a deterministic way. You can verify the issue is fixed by running the tests again, the process usually is:

run mutation testing, get the report with the escaped mutants
apply an escaped mutant manually and add a test case to capture it (i.e. add a failing test case) and then revert the escaped mutant
you can go a step further and try to remove some tests see if they are redundant or not (easier to apply on new elements of the codebase)

Are we going to integrate it into PR delivery process or run once in a while manually?

Running it once would help to raise the quality one time which is not bad but I would consider adding it to the "delivery process" process if by that you mean PRs when adding or refactoring elements.

theofidry commented 5 years ago

Looks like you can't ping other repositories teams... So:

/cc @sanmai, @sidz, @borNfreee, @BackEndTea

BackEndTea commented 5 years ago

For which kinds of tests mutation testing makes sense (integration, MFTF, web API, etc) Tests execution time implications Considering the size of Magento codebase, we might only need to "infest" select modules while running tests for these modules. We need to think about other possible optimizations

Infection can take a lot of time, especially if tests can't be ran in paralel, due to reliance on the state of a database or filesystem. Therefore i think it may be best to integrate infection in important modules first, and only on tests that are fast, and can be ran in parallel. So if its easy to do, take out big e2e tests.

Now if we can manage that, then we can consider adding bigger tests and see how infection preforms.

Is it possible to get repeatable results? How do we verify that the fixed issue?

I think @theofidry explained this well.

Are we going to integrate it into PR delivery process or run once in a while manually?

Generally speaking, if something like this isn't automated, then it won't be ran often. You can only run it on changed code, which means there isn't a big impact on the PR time.

larsroettig commented 5 years ago

@paliarush maybe it is a good idea to start with only PHPUnit Test and use for Travis only the changes from the PR( https://backendtea.github.io/2018/06/10/test-your-tests-are-testing/#usage).

Next Step is for me to have a night build or something like that to find bugs in the core modules.

What do you mean it this possible to that?

sanmai commented 5 years ago

Infection is not just a theoretical tool, but a very practical. Best would be to start with a PoC run, to show practical test deficiencies. Or even bugs: sometime, but not often, you can find that that's not the test broken, but there's an actual bug.

sdzhepa commented 5 years ago

Hello @larsroettig

According to the label "Feature Request" this ticket has been moved into "community-features" track

Thank you for collaboration

magento / community-features