The Move Mutator is a tool that mutates Move source code. It can help test the robustness of Move specifications and tests by generating different code versions (mutants). The tool mutates only the source code. Unit tests and specification blocks are not affected by the tool.
The Move Specification Tester is a tool used to test the quality of the Move specifications. How it works:
If some mutants are not killed, it means that the specification has issues and is incorrect or not tight enough to catch such cases, so it should be improved.
The Move Mutation Tester is a tool used to test the quality of the test suite and the source code. How it works:
If the mutants are not killed, it might indicate the quality of the test suite could be improved, or in some rare cases, it might indicate an error in the original source code.
To build the tools, run:
$ cargo install --git https://github.com/eigerco/move-spec-testing.git --locked move-mutator
$ cargo install --git https://github.com/eigerco/move-spec-testing.git --locked move-spec-test
$ RUSTFLAGS="--cfg tokio_unstable" cargo install --git https://github.com/eigerco/move-spec-testing.git --locked move-mutation-test
That will install the tools into ~/.cargo/bin
directory (at least on MacOS and Linux).
Ensure to have this path in your PATH
environment. This step can be done with the below command.
$ export PATH=~/.cargo/bin:$PATH
To uninstall tools, run:
$ cargo uninstall move-mutator
$ cargo uninstall move-spec-test
$ cargo uninstall move-mutation-test
The basic tool overview is shown in the below chapters. To dive more deeply into each tool, please check out the documentation here:
All tools respect the RUST_LOG
variable, and it will print out as much information as the variable allows.
There is possibility to enable logging only for the specific modules.
Please refer to the env_logger documentation for more details.
To run these tools, example projects shall be used that are provided here.
To start specification testing, run the following command from the repo directory:
$ move-spec-test -p move-mutator/tests/move-assets/same_names
The generated output should be very similar to the following (there may also be some additional Prover logs visible):
Total mutants tested: 4
Total mutants killed: 4
╭────────────────────────────────────────────────┬────────────────┬────────────────┬────────────╮
│ Module │ Mutants tested │ Mutants killed │ Percentage │
├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤
│ sources/m1/m1_1/Negation.move::Negation_m1_1 │ 1 │ 1 │ 100.00% │
├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤
│ sources/m2/Negation.move::Negation_m2 │ 1 │ 1 │ 100.00% │
├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤
│ sources/m1/Negation.move::Negation_m1 │ 1 │ 1 │ 100.00% │
├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤
│ sources/Negation.move::Negation_main │ 1 │ 1 │ 100.00% │
╰────────────────────────────────────────────────┴────────────────┴────────────────┴────────────╯
To generate a report in a JSON format, use the -o
option:
$ move-spec-test -p move-mutator/tests/move-assets/poor_spec -o report.json
The sample report generated for the above command will look as follows:
{
"files": {
"sources/Sum.move": [
{
"module": "Sum",
"tested": 4,
"killed": 0,
"mutants_alive_diffs": [
"--- original\n+++ modified\n@@ -1,6 +1,6 @@\n module TestAccount::Sum {\n fun sum(x: u128, y: u128): u128 {\n- let sum_r = x + y;\n+ let sum_r = x - y;\n\n spec {\n // Senseless specification - mutator will change + operator to -*/ but spec won't notice it.\n",
"--- original\n+++ modified\n@@ -1,6 +1,6 @@\n module TestAccount::Sum {\n fun sum(x: u128, y: u128): u128 {\n- let sum_r = x + y;\n+ let sum_r = x * y;\n\n spec {\n // Senseless specification - mutator will change + operator to -*/ but spec won't notice it.\n",
"--- original\n+++ modified\n@@ -1,6 +1,6 @@\n module TestAccount::Sum {\n fun sum(x: u128, y: u128): u128 {\n- let sum_r = x + y;\n+ let sum_r = x / y;\n\n spec {\n // Senseless specification - mutator will change + operator to -*/ but spec won't notice it.\n",
"--- original\n+++ modified\n@@ -1,6 +1,6 @@\n module TestAccount::Sum {\n fun sum(x: u128, y: u128): u128 {\n- let sum_r = x + y;\n+ let sum_r = x % y;\n\n spec {\n // Senseless specification - mutator will change + operator to -*/ but spec won't notice it.\n"
]
}
]
}
}
To check possible options, run:
$ move-spec-test --help
To start mutation test, run the following command from the repo directory:
$ move-mutation-test --package-dir move-mutator/tests/move-assets/simple -o report.txt
A shortened sample output for the above command will look as follows:
Total mutants tested: 229
Total mutants killed: 203
╭────────────────────────────────────────────────┬────────────────┬────────────────┬────────────╮
│ Module │ Mutants tested │ Mutants killed │ Percentage │
├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤
│ sources/Operators.move::Operators::and │ 3 │ 2 │ 66.67% │
├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤
│ sources/Operators.move::Operators::div │ 5 │ 5 │ 100.00% │
├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤
│ sources/Operators.move::Operators::eq │ 5 │ 5 │ 100.00% │
├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤
│ sources/Operators.move::Operators::gt │ 6 │ 6 │ 100.00% │
├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤
│ sources/StillSimple.move::StillSimple::sample6 │ 30 │ 30 │ 100.00% │
├────────────────────────────────────────────────┼────────────────┼────────────────┼────────────┤
│ sources/Sum.move::Sum::sum │ 4 │ 4 │ 100.00% │
╰────────────────────────────────────────────────┴────────────────┴────────────────┴────────────╯
The sample report.txt
generated for the above command will look as follows:
{
"files": {
"sources/Operators.move": [
{
"module_func": "Operators::and",
"tested": 3,
"killed": 2,
"mutants_alive_diffs": [
"--- original\n+++ modified\n@@ -108,7 +108,7 @@\n }\n\n fun and(x: u64, y: u64): u64 {\n- x & y\n+ y&x\n }\n\n // Info: we won't kill a mutant that swaps places (false-positive)\n"
]
},
{
"module_func": "Operators::div",
"tested": 5,
"killed": 5,
"mutants_alive_diffs": []
},
[...]
To explore other tool options, run:
$ move-mutation-test --help
Move Specification tool runs Move Mutator internally, however there is a possibility to run it manually.
To generate mutants for all files within a test project (for the whole Move package), run:
$ move-mutator -p move-mutator/tests/move-assets/simple/
By default, the output shall be stored in the mutants_output
directory unless otherwise specified.
To check possible options, run:
$ move-mutator --help
All tools in this repo are released under the open source Apache License
We are engineers. We contribute to various ecosystems by building low level implementations and core components. We built these tools because we believe in Move. We are happy to contribute to the Aptos ecosystem and will continue to do so in the future.
Contact us at hello@eiger.co Follow us on X/Twitter