Discovery: implement Patch-Only Coverage for Feature Branches (in cases of carryforward not forwarded)

codecovdesign commented 2 weeks ago

Context: longer description here, tldr: in large monorepos, users often do not run every test on every commit. This can cause issues with accurate coverage reporting, especially when carryforward flags don't run/forward. Goal here is to have an iteration that removes potentially false reporting from report

Problem

Since not all tests are run consistently and carryforward flags may not always apply, the coverage data for changes and the overall project might be inaccurate on feature branches.

Solution

Explore a "patch-only coverage" option for feature branches. This would allow users to focus on coverage for the changes in a specific patch rather than the entire project, providing a more accurate representation of coverage for feature branches in scenarios where not all tests are run.

Initial iteration: default to display patch only summary when carryforward flags are not forwarded/ran
- default path: (conditions to show 1, full list of uploads have not successfully processed for a given commit, and 2, Reports not carried forward)
  - patch only summary on feature branches
  - project+patch summary on main
Longer term: consider opt-in/out for orgs/repos that would prefer to see patch only anyway
- both: continue to align and make the PR comment and app display consistent
- configurable path:
  - project and patch summary on all branches
  - the user should be able to specify they want this

summary

WIP: designs

rohan-at-sentry commented 2 weeks ago

@codecovdesign @aj-codecov did we align on a roll-out strategy on the call we had earlier this week? I know I was pushing for feature flagging and turning this on for specific orgs, but not sure where the conversation landed after I had to leave

codecovdesign commented 2 weeks ago

where the conversation landed

@rohan-at-sentry Jerrod wanted to run this by @eliatcodecov

EDIT: quick review with eli today and feedback was to better contextualize the data. updated the designs accordingly:

:information_source: patch only coverage is shown because not all tests are run on this branch.

codecovdesign commented 3 days ago

sync on 9/10

@calvin-codecov to investigate and/or implement this iteration
@Adal3n3 incorporating / in tandem with ongoing discovery about processing states: https://github.com/codecov/engineering-team/issues/2220

codecov / engineering-team