Summary:
Enhance Codecov's reporting capabilities to ensure customers can verify that patch coverage for a given pull request (PR) is calculated correctly. This feature will provide clear visibility into whether all files present in the Git diff for a PR have 1) received a coverage report and 2) been successfully processed.
Problem Statement:
Many of our customers primarily focus on patch coverage rather than the overall monorepo coverage. Currently, it is challenging to validate whether patch coverage is being calculated accurately, because the customer doesn't know if they missed an upload due to internal sharding of test suites. There is no straightforward way to confirm if all files in the Git diff for a PR are accounted for in the coverage report and whether the report has been processed correctly. This lack of clarity can lead to mistrust in the coverage metrics or missed opportunities to address gaps in test coverage.
Proposed Solution:
Introduce a feature that allows users to verify the following for a given PR:
File Coverage Status: Identify all files in the Git diff for the PR and check if they have been included in the coverage report.
Processing Confirmation: Confirm that the coverage report for each file has been successfully processed without errors.
Detailed Feedback: Provide a detailed summary or diagnostic view highlighting any missing files, unprocessed reports, or other anomalies affecting patch coverage.
This could be implemented as a new section in the PR comment or a standalone endpoint in the Codecov API.
Use Case(s):
A customer submits a PR and wants to ensure that all files in the Git diff have test coverage accounted for.
Debugging cases where patch coverage seems unexpectedly low due to missing or unprocessed reports.
Teams working with monorepos who need to focus on specific patches without being distracted by irrelevant coverage data.
Benefits:
Increased confidence in the accuracy of patch coverage calculations.
Faster debugging and resolution of missing or unprocessed coverage reports.
Improved user experience for teams focused on patch coverage in large monorepos.
Reduced support burden as customers can self-diagnose issues related to patch coverage.
Additional Context:
This aligns with customer feedback requesting more transparency in how patch coverage is calculated and displayed.
The feature could integrate seamlessly into existing PR comments or act as a separate verification tool to complement Codecov's current functionality.
Summary:
Enhance Codecov's reporting capabilities to ensure customers can verify that patch coverage for a given pull request (PR) is calculated correctly. This feature will provide clear visibility into whether all files present in the Git diff for a PR have 1) received a coverage report and 2) been successfully processed.
Problem Statement:
Many of our customers primarily focus on patch coverage rather than the overall monorepo coverage. Currently, it is challenging to validate whether patch coverage is being calculated accurately, because the customer doesn't know if they missed an upload due to internal sharding of test suites. There is no straightforward way to confirm if all files in the Git diff for a PR are accounted for in the coverage report and whether the report has been processed correctly. This lack of clarity can lead to mistrust in the coverage metrics or missed opportunities to address gaps in test coverage.
Proposed Solution:
Introduce a feature that allows users to verify the following for a given PR:
This could be implemented as a new section in the PR comment or a standalone endpoint in the Codecov API.
Use Case(s):
Benefits:
Additional Context: