Missing output of "Apex Code Coverage by Class" when "Store Only Aggregated Code Coverage" is enabled in the Test Execution settings

[amtrack]

Summary

sf apex run test --code-coverage --wait 10 does not output the Apex Code Coverage by Class when "Store Only Aggregated Code Coverage" is enabled in the Test Execution settings (apexSettings > enableAggregateCodeCoverageOnly > true).

Steps To Reproduce:

Please see this Git repository reproducing this issue using GitHub Actions: https://github.com/mdapi-issues/mre-missing-output-for-aggregated-only-coverage. To reproduce manually, see the instructions.

Expected result

The output after === Apex Code Coverage by Class should look like:

=== Apex Code Coverage by Class
CLASSES  PERCENT  UNCOVERED LINES
───────  ───────  ───────────────
Greeter  100%

Actual result

The output after === Apex Code Coverage by Class is empty:

=== Apex Code Coverage by Class

Additional information

=== Test Summary
NAME                 VALUE
───────────────────  ─────────────────────────────
Outcome              Passed
Tests Ran            1
Pass Rate            100%
Fail Rate            0%
Skip Rate            0%
Test Run Id          7077Y00004lsieX
Test Execution Time  9 ms
Org Id               00D7Y000000GND2UAO
Username             test-8nhp9pk7g82s@example.com
Org Wide Coverage    100%

=== Test Results
TEST NAME           OUTCOME  MESSAGE  RUNTIME (MS)
──────────────────  ───────  ───────  ────────────
Test_Greeter.greet  Pass              9

=== Apex Code Coverage by Class
- CLASSES  PERCENT  UNCOVERED LINES
- ───────  ───────  ───────────────
- Greeter  100%

Salesforce Extension Version in VS Code: -

Salesforce CLI Version: @salesforce/cli/2.41.8 darwin-arm64 node-v20.10.0

OS and version: MacOS Sonoma 14.5

VS Code version: -

Most recent version of the extensions where this was working: -

[git2gus[bot]]

This issue has been linked to a new work item: W-15783639

[daphne-sfdc]

Hi @amtrack, Thank you for filing this issue. Yes, this is a bug, and I was able to reproduce it in my environment with a large project containing 4000 Apex classes and 4000 Apex test classes and the dreamhouse-lwc sample project. As my team member @peternhale responded to your previous issue, we have work in our backlog for handling large Apex test runs, and we will work on this bug as part of that effort.

Thank you for your patience. We'll keep you updated when we have more progress on resolving your issue.

[amtrack]

@daphne-sfdc Just to double-check, this ticket here isn't related to large codebases, it is only related to the Test Execution setting "Store Only Aggregated Code Coverage". The reproduction has only 2 ApexClasses, 1 is a test class. I think this could be a quick fix.

10:41:26.843] DEBUG (sf:elapsedTime): CodeCoverage.getAggregateCodeCoverage - enter
[10:41:26.843] DEBUG (sf:elapsedTime): CodeCoverage.getAggregateCodeCoverage - exit
    elapsedTime: 0.176333

Maybe the getAggregateCodeCoverage is called without apex class ids and returns immediately with an empty response: https://github.com/forcedotcom/salesforcedx-apex/blob/b3ede186ad57af393ef2d200bdadaebe2ada3c84/src/tests/codeCoverage.ts#L102

[daphne-sfdc]

Hi @amtrack,

I spoke to my team today and received confirmation that this is working as designed. I spent some time to study the code and run tests on it in dreamhouse-lwc - thank you for pointing me to the getAggregateCodeCoverage() function - and discovered this is not an easy fix. This is how the code coverage currently works:

After the Apex test run completes, the results are passed into formatAsyncResults() in asyncTests.ts. If you ran the tests with the --code-coverage flag, the check in Line 258 is True, and the code for formatting code coverage is run. If you run the Apex tests with the "Store Only Aggregated Code Coverage" box checked in the Apex Test Execution page, the perClassCovMap is set to an empty map, which means coveredApexClassIdSet is an empty set, as the values are populated to coveredApexClassIdSet during the calculation of per class code coverage. coveredApexClassIdSet is passed as parameter into getAggregateCodeCoverage(). In Line 107, we have the if (apexClassIdSet.size === 0) { validation, which returns empty code coverage results if apexClassIdSet is empty, which is True in this case. That's why the "Apex Code Coverage by Class" section of the Apex test results doesn't have a table.

The Apex classes listed in the "Apex Code Coverage by Class" table are the classes that are covered by the Apex tests in the current test run. That set is identified by the calculation of per class code coverage and cannot be inferred otherwise from the Apex test run. One alternative is to list all Apex classes present in the org in that table, since the "Store Only Aggregated Code Coverage" setting is designed to be used for large test runs, which often refers to all the Apex test classes in a given org. However, that would limit the results from when that setting is checked to only make sense in one use case.

Our team does agree that while the feature is working as originally designed, it's not intuitive, and thus we are currently investigating this issue to understand the alternatives for displaying useful information in the "Apex Code Coverage by Class" table. In the meantime, please continue to use your workaround, and we will keep you updated with our findings. Thank you for your patience.

[daphne-sfdc]

Hi @amtrack, We have some good news for you - we have a fix for this issue! Now when the "Store Only Aggregated Code Coverage" setting is checked, then all the entries in the ApexCodeCoverageAggregate Tooling API table will be displayed in the Apex test results - more details in the PR linked above.

To get this fix into your CLI now, run sf plugins install apex@3.1.15 in your Terminal to install the version of plugin-apex that got released with this fix. If you would like to use this in the Salesforce Extensions for VSCode, we have done the dependency update and it will be included in next week's release.

Please let us know if this works for you and I can close this issue.

[amtrack]

@daphne-sfdc Thank you very much! 🥳 Yes, I confirm this works for us with apex@3.1.15.

[daphne-sfdc]

@amtrack Thanks for the confirmation, and I'm very happy to hear that we're able to resolve your issue. 😀

Closing this issue as completed. ✅