Add mutation testing (code quality)

kabalin commented 6 months ago

This issue is about exploring possibility of adding Mutation testing support, which reports tests effectiveness. This information can be used to improve both code and tests quality.

Short summary of principle underneath mutation testing (from Stryker):

Mutation testing introduces changes to your code, then runs your unit tests against the changed code. It is expected that your unit tests will now fail. If they don't fail, it might indicate your tests do not sufficiently cover the code.

Bugs, or mutants, are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are.

It supports composer installation, this makes it simple to add as dependency, it also depends on pcov|xdebug (or alternatively needs coverage.xml from phpunit run, but using debugger probably easier).

The brief outline of the task:

[ ] Make it work locally for plugin testing, possibly along with moodle-plugin-ci installed for CLI run.
[ ] Add command class:
- infection run requires json configuration file that needs to be generated for the plugin if it does not exist.
- support of CLI options
[ ] Add tests for Infection command.
[ ] Add documentation.
[ ] Investigate reporting to Stryker, so that one can see how code quality improves over time 🚀

ewallah commented 3 months ago

Any leads how this can be already done without moode-plugin-ci? Perhaps this will shed some light how it can be implemented.

kabalin commented 3 months ago

Any leads how this can be already done without moode-plugin-ci? Perhaps this will shed some light how it can be implemented.

That is what step 1 in outline is about, to explore without moode-plugin-ci changes :) Feel free to do it and share outcomes 😉

ewallah commented 3 months ago

Finally, I got my first infection output:

php infection.phar -vvv --debug --threads=max --only-covered --ignore-msi-with-no-mutations --min-covered-msi=100

Box Requirements Checker
========================

> Using PHP 8.2.17
> PHP is using the following php.ini file:
  /etc/php/8.2/cli/php.ini

> Checking Box requirements:
  ✔ The application requires the version "^8.1" or greater.
  ✔ The application requires the extension "zlib".
  ✔ The application requires the extension "dom".
  ✔ The application requires the extension "json".
  ✔ The package "colinodell/json5" requires the extension "json".
  ✔ The application requires the extension "libxml".
  ✔ The package "nikic/php-parser" requires the extension "tokenizer".

 [OK] Your system is ready to run the application.

[debug] Checking INFECTION_ALLOW_XDEBUG
[debug] The Xdebug extension is not loaded

    ____      ____          __  _
   /  _/___  / __/__  _____/ /_(_)___  ____
   / // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
 _/ // / / / __/  __/ /__/ /_/ / /_/ / / / /
/___/_/ /_/_/  \___/\___/\__/_/\____/_/ /_/

#StandWithUkraine

Infection - PHP Mutation Testing Framework version 0.27.0

[notice] You are running Infection with PCOV enabled.

Running initial test suite...

PHPUnit version: 9.5.28

   13 [============================] 3 secs

Generate mutants...

Processing source code files: 3/3
.: killed, M: escaped, U: uncovered, E: fatal error, X: syntax error, T: timed out, S: skipped, I: ignored

.EE.EEEEEE......EEEEEEEEEEEEEEEEEEEEEEEE             (40 / 40)

40 mutations were generated:
       8 mutants were killed
       0 mutants were configured to be ignored
       0 mutants were not covered by tests
       0 covered mutants were not detected
      32 errors were encountered
       0 syntax errors were encountered
       0 time outs were encountered
       0 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 100%
         Mutation Code Coverage: 100%
         Covered Code MSI: 100%

Generated Reports:
         - out.html

Please note that some mutants will inevitably be harmless (i.e. false positives).

Time: 13s. Memory: 0.06GB. Threads: 15

Moodle 4.2.6+ (Build: 20240322)
plugin checked: moodle-availability_language

To make this work, I had to:

Ensure that the tests are run randomly (executionOrder="random")
Allow junit output.
Work with covered tests only
Create a custom bootstrap file that loads vendor/autoload.php before lib/phpunit/bootstrap.php. Not the perfect solution, but now at least I see output and tests are run. With the moodle bootstrap file only, the tests hang. Without the autoload file the classes are not found ([ReflectionException (-1)] Class "availability_language\privacy\provider" does not exist).
Learn to live with the fact that my phpunit tables are broken after the run (Can not use database for testing, try different prefix) This issue has to be fixed first, how many prefixes can one invent before you kill your own database?

kabalin commented 3 months ago

Thanks @ewallah, interesting. Re point 5, that might be because of concurrency, try with --threads=1, according to docs each thread needs separate DB prefix, but it is not possible to initialise more than one using standard approach. In theory you can make config.php respect TEST_TOKEN and init phpunit for each prefix separately, then execute infection.phar with more than one thread.

This made me think, to what extent multithreaded run of phpunit could be of use for Moodle? Implementation of it exists https://github.com/paratestphp/paratest

ewallah commented 1 month ago

I was able to run infection tests with GitHub Actions:

37 mutations were generated:
      35 mutants were killed
       0 mutants were configured to be ignored
       0 mutants were not covered by tests
       2 covered mutants were not detected
       0 errors were encountered
       0 syntax errors were encountered
       0 time outs were encountered
       0 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 94%
         Mutation Code Coverage: 100%
         Covered Code MSI: 94%

To run these tests I added 2 files:

.infection.json configuration file configurable per plugin (so mutants can be enabled and disabled)
infection.yml loading infection as a PHP tool, and reordering of the autoload

P.S. The broken phpunit tables were related to a defective plugin.

moodlehq / moodle-plugin-ci

Add mutation testing (code quality) #265