Add Mutation Coverage based on Covered code to the HTML Report

[vivganes]

The metric mutationScoreBasedOnCoveredCode is already calculated by Stryker and is present in the metrics object. This is a very useful metric because it says "Your tests have covered the mutated code. But they are not useful"

I have see other mutation testing tools directly showing the same number as "Test Strength" in their HTML test report (refer PIT for Java).

Showing this number in our report tables will help the teams to improve their already existing tests, that seem to cover the mutation. This can be set as a goal by the teams and can be monitored over time.

This PR shows this additional metric in the HTML report.

[nx-cloud[bot]]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 4eee54ae3d77a72bd29d900cf86c3fd831a98d64. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

---

✅ Successfully ran 2 targets

- [`nx affected --target=lint:prettier,lint,build,test`](https://cloud.nx.app/runs/Lxpgp9bpwP?utm_source=pull-request&utm_medium=comment) - [`nx run-many --target=stryker --projects=metrics,elements,real-time -- --incremental`](https://cloud.nx.app/runs/0NwBDLbC1x?utm_source=pull-request&utm_medium=comment)

---

Sent with 💌 from NxCloud.

[hugo-vrijswijk]

Very nice. Since this is a "newer" optional metric, do you think it makes sense to hide this for reports that haven't added coverage? Or would that be the same score as the mutation score? In that case, it might make sense to hide it?

On the website, we have a different description for this metric. It should probably be the same everywhere. I also think 'Test strength' implies the effectiveness of individual tests, which is something different to this metric

[vivganes]

Very nice. Since this is a "newer" optional metric, do you think it makes sense to hide this for reports that haven't added coverage? Or would that be the same score as the mutation score? In that case, it might make sense to hide it?

On the website, we have a different description for this metric. It should probably be the same everywhere. I also think 'Test strength' implies the effectiveness of individual tests, which is something different to this metric

Thanks for your suggestion, @hugo-vrijswijk

1. When you say "reports that haven't added coverage", do you mean the case where code coverage is 0%?

2. Also, when we add "Mutation score based on covered code" as the column title, it could lead to the table header looking a little more distorted due to the long column header. I agree 'test strength' does not exactly convey the real meaning. I am confused what to add as a column header here.

[hugo-vrijswijk]

1. When you say "reports that haven't added coverage", do you mean the case where code coverage is 0%?

I mean where the mutation score and the mutation score based on coverage is the same. When there's no NoCoverage mutants. Then it's just duplicate data.

2. Also, when we add "Mutation score based on covered code" as the column title,  it could lead to the table header looking a little more distorted due to the long column header.  I agree 'test strength' does not exactly convey the real meaning.  I am confused what to add as a column header here.

Maybe "covered score"? The full name is too long to put in the table header, I think. Maybe @nicojs or @rouke-broersma has a good idea?

[rouke-broersma]

Naming is hard. Covered Score sounds too much like it's code coverage. I think the definition should be changed as well because it mentions code coverage, but it has nothing to do with code coverage. It should probably rather say mutant coverage?

[vivganes]

How about "Covered Mutants Kill %"? This is not very short but not very long either.

[rouke-broersma]

How about "Covered Mutants Kill %"? This is not very short but not very long either.

How about we subdivide the mutation score column horizontally and add a total, and of covered as sub column.

So it would be:

      mutation score
    total   | of covered
      50%   |   90% 

@hugo-vrijswijk

[vivganes]

How about "Covered Mutants Kill %"? This is not very short but not very long either.

How about we subdivide the mutation score column horizontally and add a total, and of covered as sub column.

So it would be:

      mutation score
    total   | of covered
      50%   |   90% 

@hugo-vrijswijk

Something like this?

[vivganes]

@hugo-vrijswijk @rouke-broersma I have updated the columns as suggested here 👇

How about "Covered Mutants Kill %"? This is not very short but not very long either.

How about we subdivide the mutation score column horizontally and add a total, and of covered as sub column. So it would be:

      mutation score
    total   | of covered
      50%   |   90% 

@hugo-vrijswijk

Something like this?

[nicojs]

I like the horizontal approach. Good idea 👍🏻. However, I would like to see "total" to the right, "of covered" to the left. That way the focus remains on the total instead of covered. Maybe also name it "of total" instead of " total" 🤔

[rouke-broersma]

I like the horizontal approach. Good idea 👍🏻. However, I would like to see "total" to the right, "of covered" to the left. That way the focus remains on the total instead of covered. Maybe also name it "of total" instead of " total" 🤔

I get what you mean but if we put it to the right I think that signals that it's less important. It would no longer be the first number in the column so it won't be the first thing you see anymore.

[vivganes]

@nicojs @rouke-broersma - I managed to implement a combination of suggestions from both of you. This code is now in my branch.

[vivganes]

I have updated the screenshots both in windows and linux from my local machines. The tests are passing in my local machine. I am not sure how to debug why the screenshot comparisons are failing in the CI system.

Any tips here will be helpful to get this to ready state.

@rouke-broersma @nicojs

[hugo-vrijswijk]

You can download the failing screenshots and update them manually. There's also an action that updates them. I'll run it now, but I'm not sure if it will work to push to your repository

Edit: yeah I cannot start it for your branch unfortunately I've started the screenshot update action on a separate branch that I can merge into yours when it finishes

[vivganes]

You can download the failing screenshots and update them manually. There's also an action that updates them. I'll run it now, but I'm not sure if it will work to push to your repository

Edit: yeah I cannot start it for your branch unfortunately

@hugo-vrijswijk Sorry for asking a newbie question. how do I download the screenshots? I am assuming it is from the link in the logs. Can you please confirm?

[hugo-vrijswijk]

I like the horizontal approach. Good idea 👍🏻. However, I would like to see "total" to the right, "of covered" to the left. That way the focus remains on the total instead of covered. Maybe also name it "of total" instead of " total" 🤔

I get what you mean but if we put it to the right I think that signals that it's less important. It would no longer be the first number in the column so it won't be the first thing you see anymore.

I do feel like for most left-to-right readers the number on the right feels like the most important one. It's usually where the summation is. We also show the total number of mutants as the right-most value. If we want to push the "Total" as the more important metric, having it on the right side makes more sense to me

[hugo-vrijswijk]

You can download the failing screenshots and update them manually. There's also an action that updates them. I'll run it now, but I'm not sure if it will work to push to your repository Edit: yeah I cannot start it for your branch unfortunately

@hugo-vrijswijk Sorry for asking a newbie question. how do I download the screenshots? I am assuming it is from the link in the logs. Can you please confirm?

I think that would work. Or at the workflow overview there is a link at the bottom:

I've pushed the new images now though

[bundlemon[bot]]

BundleMon (elements)

Files updated (3)

Status | Path | Size | Limits :------------: | ------------ | :------------: | :------------: :white_check_mark: | index.js
| 269.24KB (+559B +0.2%) | - :white_check_mark: | index.cjs
| 198.81KB (+457B +0.22%) | - :white_check_mark: | mutation-test-elements.js
| 198.81KB (+457B +0.22%) | -

Total files change +1.44KB +0.22%

Final result: :white_check_mark:

View report in BundleMon website ➡️

---

Current branch size history | Target branch size history

[rouke-broersma]

I like the horizontal approach. Good idea 👍🏻. However, I would like to see "total" to the right, "of covered" to the left. That way the focus remains on the total instead of covered. Maybe also name it "of total" instead of " total" 🤔

I get what you mean but if we put it to the right I think that signals that it's less important. It would no longer be the first number in the column so it won't be the first thing you see anymore.

I do feel like for most left-to-right readers the number on the right feels like the most important one. It's usually where the summation is. We also show the total number of mutants as the right-most value. If we want to push the "Total" as the more important metric, having it on the right side makes more sense to me

Yes but it's not on on the right, it's in second position. So it's neither the most left nor the most right.

[hugo-vrijswijk]

But it is the right-most of the scores.

I don't have a super strong opinion on this. If other people prefer Total - Score that's fine by me

[vivganes]

So far, the total score is shown just after the file/directory name.

For the users who would have mentally conditioned themselves to quickly scan the numbers immediately after the file name, having "total" in the left makes sense. This way, nothing is changed for them.

Also, if they suddenly see a very less number in the visual space they expect, they may get confused also for sometime.

There is a danger of them thinking that their team has significantly improved the numbers recently, which is not the case.