Code Coverage is falsely calculated in SonarQube

[xL3o]

I first wanted to thank you for providing this plugin! It does exactly what it should.

However, once I import the code coverage report to SonarQube's generic test coverage report path (sonar.coverageReportPaths), it shows the uncovered lines but the code coverage stays at 0%.

Example:

The apex unit tests cover all lines of the SP_Access...cls file except 70-75. This is also shown in SonarQube:

Nevertheless, the Code Coverage for that class is 0%.

Is it a SonarQube configuration issue, or does the code coverage report need to contain all covered lines also?

[mcarvin8]

Thanks @xL3o for reporting this issue! Appreciate seeing feedback.

When I initially developed this plugin and added all lines (covered and uncovered) to the XML, I noticed that the Salesforce CLI output (JSON) contained extra line numbers as "covered" which weren't in the file. For example, it would report Line 100 as "covered" when there was only 98 lines of code in the class. I haven't tested this in a bit, so I'm unclear if it's fixed or reproducible by other users. Before I raise a Salesforce CLI bug on Salesforce's repo, I would like to do the following:

1. Create a Pre-Release version which adds covered and uncovered lines to the XML.
2. Have you download the pre-release version on your end and see if you can get SonarQube to accept the file. It will not accept it if the CLI is reporting Out-of-Range lines as "Covered".

If you can replicate that error, I'll raise a bug with the Salesforce CLI team.

If you can't replicate that error, I can release a new version of this plugin with those updates.

[mcarvin8]

In terms of why SQ is not updating the Code Coverage metric in the 1 file explorer view, let me check on my end if I can replicate that.

I know I have confirmed coverage when looking at the file directly. I can't recall at the moment if the other view displayed coverage %.

[mcarvin8]

@xL3o - The pre-release/beta build which will add all lines (covered and uncovered) to the XML has been published.

sf plugins install apex-code-coverage-transformer@1.4.1-beta.1

Please test and let me know how that works out for you when you get a moment.

[xL3o]

@mcarvin8 that did not quite work. Now for files that contain uncovered lines, the report always starts with the first uncovered line and adds covered lines below it: Those lines are also out-of-range now:

[xL3o]

@mcarvin8 that did not quite work. Now for files that contain uncovered lines, the report always starts with the first uncovered line and adds covered lines below it: Those lines are also out-of-range now:

But the covered:uncovered ratio is right. 5 Uncovered and 30 Covered. Just the Covered line numbers are incorrect.

[xL3o]

I believe you can open a bug with the salesforce CLI team:

The class has only 80 lines but the .json reports >80 line numbers...

[mcarvin8]

https://github.com/forcedotcom/cli/issues/1568

@mshanemc - Seems like the above issue on the CLI repository explains the current issue we are seeing while troubleshooting this plugin. Out-Of-Range lines are being generated in the JSON file by the latest Salesforce CLI in most scenarios, which causes SonarQube to fail to parse the code coverage XML created by this plugin (this plugin just converts the JSON to an XML SonarQube will accept).

I don't know if you would like me to create a new bug in the CLI, but you closed the above since it's working as "intended" but you had a longer-term plan to correct this behavior.

In the latest release of this plugin, I only add "uncovered" lines to the XML file. But I'm trying to troubleshoot this issue @xL3o is seeing in SonarQube, so I decided to release a "beta" build which adds covered and uncovered lines to the XML.

Side-Note: I'm suspecting this issue could stem from something in the SQ configuration if we keep the current logic intact on the main branch (only add uncovered lines to the XML which is accurate).

[mcarvin8]

Tests Done So Far:

SonarQube Version - Enterprise EditionVersion 9.9.3 (build 79811)

Test 1 - apex-code-coverage-transformer@1.4.1-beta.1 -Beta build which has covered and uncovered lines.

Salesforce CLI command - sf project deploy validate -l RunSpecifiedTests -t PrepareMySandboxTest --coverage-formatters json --results-dir coverage -x manifest/package.xml -w 240 --verbose

Plugin command - sf apex-code-coverage transformer transform -j "coverage/coverage/coverage.json"

Sonar CLI command - sonar-scanner -Dsonar.qualitygate.wait=$SONAR_GATE -Dsonar.projectKey=$CI_PROJECT_PATH_SLUG -Dsonar.gitlab.ref_name=$SONAR_REF -Dsonar.gitlab.commit_sha=$CI_COMMIT_SHORT_SHA -Dsonar.coverageReportPaths=coverage.xml

The XML created by the plugin, which has less lines of code than the actual file does. But the Uncovered Lines are correct.

<?xml version="1.0"?>
<coverage version="1">
    <file path="force-app/main/default/classes/PrepareMySandbox.cls">
        <lineToCover lineNumber="7" covered="false"/>
        <lineToCover lineNumber="8" covered="true"/>
        <lineToCover lineNumber="9" covered="true"/>
        <lineToCover lineNumber="10" covered="true"/>
        <lineToCover lineNumber="11" covered="true"/>
        <lineToCover lineNumber="12" covered="false"/>
        <lineToCover lineNumber="13" covered="true"/>
        <lineToCover lineNumber="14" covered="true"/>
        <lineToCover lineNumber="15" covered="true"/>
        <lineToCover lineNumber="16" covered="true"/>
        <lineToCover lineNumber="17" covered="true"/>
        <lineToCover lineNumber="18" covered="true"/>
        <lineToCover lineNumber="19" covered="true"/>
        <lineToCover lineNumber="20" covered="true"/>
        <lineToCover lineNumber="21" covered="true"/>
        <lineToCover lineNumber="22" covered="true"/>
        <lineToCover lineNumber="23" covered="true"/>
        <lineToCover lineNumber="24" covered="true"/>
        <lineToCover lineNumber="25" covered="true"/>
        <lineToCover lineNumber="26" covered="true"/>
        <lineToCover lineNumber="27" covered="true"/>
        <lineToCover lineNumber="28" covered="true"/>
        <lineToCover lineNumber="29" covered="true"/>
        <lineToCover lineNumber="30" covered="true"/>
        <lineToCover lineNumber="31" covered="true"/>
        <lineToCover lineNumber="32" covered="true"/>
        <lineToCover lineNumber="33" covered="true"/>
        <lineToCover lineNumber="34" covered="true"/>
        <lineToCover lineNumber="35" covered="true"/>
        <lineToCover lineNumber="36" covered="true"/>
        <lineToCover lineNumber="37" covered="true"/>
    </file>
</coverage>

In SQ, the coverage indicators stop at the Line 37

But, I do get the Coverage % to show in the "Code" menu

[mcarvin8]

Test 2 - apex-code-coverage-transformer@1.4.0 - Latest Build which just prints Uncovered Lines Salesforce CLI command - sf project deploy validate -l RunSpecifiedTests -t PrepareMySandboxTest --coverage-formatters json --results-dir coverage -x manifest/package.xml -w 240 --verbose

Plugin command - sf apex-code-coverage transformer transform -j "coverage/coverage/coverage.json"

Sonar CLI command - sonar-scanner -Dsonar.qualitygate.wait=$SONAR_GATE -Dsonar.projectKey=$CI_PROJECT_PATH_SLUG -Dsonar.gitlab.ref_name=$SONAR_REF -Dsonar.gitlab.commit_sha=$CI_COMMIT_SHORT_SHA -Dsonar.coverageReportPaths=coverage.xml

The XML created by the plugin:

<?xml version="1.0"?>
<coverage version="1">
    <file path="force-app/main/default/classes/PrepareMySandbox.cls">
        <lineToCover lineNumber="7" covered="false"/>
        <lineToCover lineNumber="12" covered="false"/>
    </file>
</coverage>

@xL3o - I am able to replicate your issue. Uncovered lines show when looking in the file directly, but the % is not updated in the "Code" menu.

So basically @xL3o @mshanemc ,

• With the latest version of this plugin which only prints uncovered lines, the file view in SonarQube directly shows uncovered lines, but SonarQube will not update the Coverage % in other views.
• With the beta version of this plugin that prints covered and uncovered lines, the file view in SonarQube will show both and the Coverage % will populate in other menus. However, the Salesforce CLI will print either Out-of-Range lines or not enough lines.

So until the CLI is fixed to print the accurate covered lines, I might have to look into updating this plugin to count lines in the file itself maybe??

[mcarvin8]

@xL3o - I have published a new beta build which should read all of the files to get the total # of lines. If the line is not listed as "Uncovered" in the JSON file, the plugin will assume it's "Covered". I think this is a safe workaround until the Salesforce CLI is resolved to accurately report "Covered" lines.

https://github.com/mcarvin8/apex-code-coverage-transformer/compare/v1.4.1-beta.1...v1.4.1-beta.2

Please download this version, test it out, and let me know if that does the trick when you get a moment. Once you confirm this is working on your end, I'll merge the pull request and release a new official build.

sf plugins install apex-code-coverage-transformer@1.4.1-beta.2

When I tested 1.4.1-beta.2 on my end, this is working as expected:

[xL3o]

Hi @mcarvin8 , thanks for trying to build a workaround for now. However, we have to keep in mind, that Salesforce ignores comments and certain other lines. The new coverage report version now get accepted but is still calculating wrong coverage: Official SF Coverage: Report Result: `

` SQ reported Coverage:

Solution Proposal: In the json-summary file that can be generated by appending "--coverage-formatters json-summary" additionally, the ration of uncovered and covered lines is contained:

*Notice how SF themself disagree on the coverage percentage. In the deployment verbose output it is 83% and in the summary json it is 85.71%. ;)

If you could adjust the beta version in a way to just add the amount of covered lines specified in the summary and skip the other lines, we should get pretty close to the "official" code coverage percentage.

[xL3o]

Hi @mcarvin8 , thanks for trying to build a workaround for now. However, we have to keep in mind, that Salesforce ignores comments and certain other lines. The new coverage report version now get accepted but is still calculating wrong coverage: Official SF Coverage: Report Result: <file path="force-app/main/default/classes/SP_Access....cls"> <lineToCover lineNumber="1" covered="true"/> <lineToCover lineNumber="2" covered="true"/> <lineToCover lineNumber="3" covered="true"/> <lineToCover lineNumber="4" covered="true"/> <lineToCover lineNumber="5" covered="true"/> <lineToCover lineNumber="6" covered="true"/> <lineToCover lineNumber="7" covered="true"/> <lineToCover lineNumber="8" covered="true"/> <lineToCover lineNumber="9" covered="true"/> <lineToCover lineNumber="10" covered="true"/> <lineToCover lineNumber="11" covered="true"/> <lineToCover lineNumber="12" covered="true"/> <lineToCover lineNumber="13" covered="true"/> <lineToCover lineNumber="14" covered="true"/> <lineToCover lineNumber="15" covered="true"/> <lineToCover lineNumber="16" covered="true"/> <lineToCover lineNumber="17" covered="true"/> <lineToCover lineNumber="18" covered="true"/> <lineToCover lineNumber="19" covered="true"/> <lineToCover lineNumber="20" covered="true"/> <lineToCover lineNumber="21" covered="true"/> <lineToCover lineNumber="22" covered="true"/> <lineToCover lineNumber="23" covered="true"/> <lineToCover lineNumber="24" covered="true"/> <lineToCover lineNumber="25" covered="true"/> <lineToCover lineNumber="26" covered="true"/> <lineToCover lineNumber="27" covered="true"/> <lineToCover lineNumber="28" covered="true"/> <lineToCover lineNumber="29" covered="true"/> <lineToCover lineNumber="30" covered="true"/> <lineToCover lineNumber="31" covered="true"/> <lineToCover lineNumber="32" covered="true"/> <lineToCover lineNumber="33" covered="true"/> <lineToCover lineNumber="34" covered="true"/> <lineToCover lineNumber="35" covered="true"/> <lineToCover lineNumber="36" covered="true"/> <lineToCover lineNumber="37" covered="true"/> <lineToCover lineNumber="38" covered="true"/> <lineToCover lineNumber="39" covered="true"/> <lineToCover lineNumber="40" covered="true"/> <lineToCover lineNumber="41" covered="true"/> <lineToCover lineNumber="42" covered="true"/> <lineToCover lineNumber="43" covered="true"/> <lineToCover lineNumber="44" covered="true"/> <lineToCover lineNumber="45" covered="true"/> <lineToCover lineNumber="46" covered="true"/> <lineToCover lineNumber="47" covered="true"/> <lineToCover lineNumber="48" covered="true"/> <lineToCover lineNumber="49" covered="true"/> <lineToCover lineNumber="50" covered="true"/> <lineToCover lineNumber="51" covered="true"/> <lineToCover lineNumber="52" covered="true"/> <lineToCover lineNumber="53" covered="true"/> <lineToCover lineNumber="54" covered="true"/> <lineToCover lineNumber="55" covered="true"/> <lineToCover lineNumber="56" covered="true"/> <lineToCover lineNumber="57" covered="true"/> <lineToCover lineNumber="58" covered="true"/> <lineToCover lineNumber="59" covered="true"/> <lineToCover lineNumber="60" covered="true"/> <lineToCover lineNumber="61" covered="true"/> <lineToCover lineNumber="62" covered="true"/> <lineToCover lineNumber="63" covered="true"/> <lineToCover lineNumber="64" covered="true"/> <lineToCover lineNumber="65" covered="true"/> <lineToCover lineNumber="66" covered="true"/> <lineToCover lineNumber="67" covered="true"/> <lineToCover lineNumber="68" covered="true"/> <lineToCover lineNumber="69" covered="true"/> <lineToCover lineNumber="70" covered="false"/> <lineToCover lineNumber="71" covered="false"/> <lineToCover lineNumber="72" covered="true"/> <lineToCover lineNumber="73" covered="false"/> <lineToCover lineNumber="74" covered="false"/> <lineToCover lineNumber="75" covered="false"/> <lineToCover lineNumber="76" covered="true"/> <lineToCover lineNumber="77" covered="true"/> <lineToCover lineNumber="78" covered="true"/> <lineToCover lineNumber="79" covered="true"/> <lineToCover lineNumber="80" covered="true"/> </file> SQ reported Coverage:

Solution Proposal: In the json-summary file that can be generated by appending "--coverage-formatters json-summary" additionally, the ration of uncovered and covered lines is contained:

*Notice how SF themself disagree on the coverage percentage. In the deployment verbose output it is 83% and in the summary json it is 85.71%. ;)

If you could adjust the beta version in a way to just add the amount of covered lines specified in the summary and skip the other lines, we should get pretty close to the "official" code coverage percentage.

@mcarvin8 Thought about it once again, we don't need to use the json-summary even. In the normal json the covered lines are already specified at the correct amount, meaning you could just count how many "1" lines there are depicting the covered lines amount and add covered lines accordingly.

[mcarvin8]

@xL3o - Let's try this beta build out then.

sf plugins install apex-code-coverage-transformer@1.4.1-beta.3

The update I've made here is:

1. Print "Uncovered" lines in the XML first exactly as they are in the normal JSON
2. Print "Covered" lines in the XML as they are given in the normal JSON
   1. If the "Covered" line is greater than the total number of lines in the file, generate a random number that's in the file count but not already included in the "covered", "uncovered", or "random" numbers (if multiple "covered" lines are out-of-range)

I know this isn't an ideal scenario, but this should maintain the total number of Covered and Uncovered Lines in the file to ensure the Coverage % is more accurate. It just might not have accurate "covered" lines, but I think in most cases, knowing the "uncovered" lines and overall code coverage % is more important.

I'll make an update in the README (assuming this works out for you and makes sense) stating that the "uncovered" lines and the code coverage % will always be accurate, but the "covered" lines may be randomly generated due to the existing Salesforce CLI bug with how it generates "covered" lines.

[xL3o]

Hi @mcarvin8 thanks for the update. Using random numbers is in my use case extremely inefficient. The transform command now takes ages to complete (while writing this comment it is still running in the background for >25 minutes). Therefore, I can't tell if it is working (once its finished I give you an update), but I can say that this takes too long for our prod pipeline. Please remove the random number functions and just start with line 1 and set it to covered if line 1 is not uncovered and the covered line amount is not reached yet.

[xL3o]

@mcarvin8 - script took too long:

[mcarvin8]

@xL3o - Sorry for the inconvenience. Please try the new beta build which starts at line 1. This also switches the check from a do statement to a for statement, so it shouldn't hang like the previous build did. Please let me know if that solves your problem when you get a moment.

sf plugins install apex-code-coverage-transformer@1.4.1-beta.4

[xL3o]

@mcarvin8 - Thanks for the fast response. It did not quite work as I wanted it to be. Following output: `

` However, the script should stop putting covered lines once it reaches the covered lines amount by the coverage.json file. In that specific example, the SP_Access class has 5 uncovered and 30 covered lines, meaning in the end the coverage.xml file should contain the correct 5 uncovered lines (what it currently does) and 30 covered lines (which it currently doesn't have; it has 75 covered lines). So you would have to first count the lines in the .json which have a 1 (== covered) and save it as a variable (e.g. coveredLinesAmount) somewhere. Second step is to fill up the .xml with covered lines starting from 1 until it reaches the coveredLinesAmount while minding not to overwrite any uncovered lines.

[mcarvin8]

@xL3o - Sorry again for the inconvenience. This new beta build should track the # of covered lines processed to ensure it stops after it has processed the total number of covered lines as found in the JSON file.

After changing my JSON locally to have 3 out of bound "covered" lines out of a total of 29 "covered" lines, I confirmed that running this new beta build produced an XML with 29 "covered" lines with the 3 out of bound values re-numbered to a value starting at 1.

Please try this one and let me know if this resolves the issue: sf plugins install apex-code-coverage-transformer@1.4.1-beta.7

[mcarvin8]

@xL3o - Disregard previous message. I figured out the issue was the missing break condition in the for loop. After adding the break to break once the out-of-bound value was re-numbered, this fixed the issue of covered lines in the XML exceeding the total number of covered lines in the JSON. This way, I don't actually need the code to track the total # of covered lines and ensure it doesn't exceed that.

Please try 1.4.1-beta.8 - sf plugins install apex-code-coverage-transformer@1.4.1-beta.8

beta.6, beta.7, and beta.8 should produce the same coverage XML (it did on my end) so you probably won't notice the difference running either if you beat me to the punch. But beta.8 will do it more efficiently.

[xL3o]

@mcarvin8 - Thanks for all the hard work. Tested the latest beta version 8 and your script does exactly what it should, but we can't trust salesforce json coverage report. I have a class that has exactly 208 lines but in the json it reports 218 lines (uncovered + covered lines combined)...

So our assumption that we can trust at least the ratio covered:uncovered lines in the json is wrong. So I believe there is no viable work around anymore to depict the correct code coverage.

Maybe you have another idea how to resolve this issue... I can test more beta version from monday onwards.

[xL3o]

We might have to wait until salesforce fixes this cli bug.

[mcarvin8]

@xL3o - Understood. Thanks for looking into this more. Yea, if json or json-summary's covered lines can't be trusted with the current CLI, I'm afraid we're at a breaking point here with workarounds with this plugin. Unless I can figure out exactly how the code coverage JSON is created/how it reports line numbers to see why it would produce extra lines, I don't think it will be feasible to update this Plugin to support "covered" lines until the Salesforce CLI is corrected.

The workaround that another user posted here was using the sfdx-hardis plugin (https://github.com/forcedotcom/cli/issues/1568#issuecomment-1431135490) to calculate code coverage during deploys but I don't really want to pivot this plugin from the Salesforce CLI to the SFDX-Hardis plugin.

Let me post a comment on https://github.com/forcedotcom/cli/issues/1568 soon with these details, since this does convey the same issues here but this was last updated over a year ago.

[mcarvin8]

I'm marking this as completed by 1.6.0 since that adds covered lines with the obvious disclaimer in the README about out-of-range line numbers. I understand the coverage won't be perfect, but this at least will work with the constraints of the current CLI/API bug, not indefinitely hang when running, and will get coverage to calculate in SonarQube.

[mcarvin8]

Once https://github.com/forcedotcom/salesforcedx-vscode/issues/5511 is resolved by Salesforce, a new version of this plugin will be released to remove the renumbering function.

[xL3o]

will get coverage to calculate in SonarQube.

Sure it is getting some coverage, just not the correct one. It is not deviating by small percentage points from the "real" value one would get from the sf apex run test, it is sometimes deviating by >10%. If anyone wants to use that to check for code coverage in their quality gates, it might be a surprise that once trying to deploy to prod, the coverage is way below the data depict in SQ and could fail the deployment.

I would leave that issue open for transparency until it got resolved by SF.

[mcarvin8]

Understood, I'll just leave this issue open for those who are using my plugin.

https://github.com/forcedotcom/salesforcedx-vscode/issues/5511 will stay open until Salesforce resolves this on their side.

[mcarvin8]

This draft branch will need to be rebased then merged into main once Salesforce resolves https://github.com/forcedotcom/salesforcedx-vscode/issues/5511.

https://github.com/mcarvin8/apex-code-coverage-transformer/tree/draft/remove-covered-line-adjustment

[jfaderanga]

hey @mcarvin8 how're you doing, it's so great to see your plugin grows and it has more very useful feature now compare to last time, thanks heaps for this mate.

[mcarvin8]

Thanks @jfaderanga for the feedback! Hope you're doing well too.

Yea my goal at this point is to leave plugin at current version until Salesforce fixes the deployment coverage reporting bug.

Unless there's a major bug with my current plugin version or someone has a good feature suggestion to add to this.