mc-imperial / dredd

Framework for evaluating C/C++ compiler testing tools
Apache License 2.0
11 stars 3 forks source link

Avoid mutating variables that require const initialization #236

Closed JamesLee-Jones closed 2 months ago

JamesLee-Jones commented 3 months ago

We cannot mutate variable declarations that require constant initialization as Dredd's mutations occur dynamically.

Fixes: #235.

afd commented 2 months ago

@JamesLee-Jones This fails with:

2024-07-05T13:45:23.8860447Z + echo 'Found 67841 mutants when mutating the SPIR-V validator source code. Expected 67998. If Dredd changed recently, the expected value may just need to be updated, if it still looks sensible. Otherwise, there is likely a problem.'
2024-07-05T13:45:23.8861595Z + exit 1
2024-07-05T13:45:23.8862744Z Found 67841 mutants when mutating the SPIR-V validator source code. Expected 67998. If Dredd changed recently, the expected value may just need to be updated, if it still looks sensible. Otherwise, there is likely a problem.

It therefore looks as if your change is slightly reducing the number of mutants that are available when spirv-val is mutated. We should look into this. The approach would be:

After that some manual or automated test case reduction should home in on where the problem is.

JamesLee-Jones commented 2 months ago

@JamesLee-Jones This fails with:

2024-07-05T13:45:23.8860447Z + echo 'Found 67841 mutants when mutating the SPIR-V validator source code. Expected 67998. If Dredd changed recently, the expected value may just need to be updated, if it still looks sensible. Otherwise, there is likely a problem.'
2024-07-05T13:45:23.8861595Z + exit 1
2024-07-05T13:45:23.8862744Z Found 67841 mutants when mutating the SPIR-V validator source code. Expected 67998. If Dredd changed recently, the expected value may just need to be updated, if it still looks sensible. Otherwise, there is likely a problem.

It therefore looks as if your change is slightly reducing the number of mutants that are available when spirv-val is mutated. We should look into this. The approach would be:

* mutate spirv-val just like the CI does with a version of dredd before this change

* mutate a different checkout of spirv-val with dredd after this change

* diff the mutated code - this should shed light on which file contains the code that exhibits the discrepancy

After that some manual or automated test case reduction should home in on where the problem is.

From looking back at the single file test that was modified, I believe it is because the check i used catches const marked variables as well as those marked as requiring const initialization and thus no mutants are produced for expressions of the form const type = value;. If this is the case, then the test case signed_int_constants.cc that was modified by these changes (and is now not) would catch this.

afd commented 2 months ago

@JamesLee-Jones can you rebase this, please?

@JonathanFoo0523 after James has rebased, can you investigate why James's change is reducing the number of mutants that get created for spirv-val? (See cxx_apps.sh under .github/workflows for details of how spirv-val is handled.)

JamesLee-Jones commented 2 months ago

@JamesLee-Jones can you rebase this, please?

@JonathanFoo0523 after James has rebased, can you investigate why James's change is reducing the number of mutants that get created for spirv-val? (See cxx_apps.sh under .github/workflows for details of how spirv-val is handled.)

I believe this commit already resolved the issue.