Codecov is able to provide a complete coverage picture for orgs doing test selection on commits

[rohan-at-sentry]

Is your feature request related to a problem? Please describe. For organizations that run a subset of tests on every commit, we sometimes do not receive a full coverage report. We sort of solve it with Carry Forward Flags.

Carry Forward Flags can have performance issues at scale, and so any solution that works for larger companies cannot reliably depend onf Carry Forward Flag implementations. This limits the usefulness of Codecov for larger companies as a result (because we cannot reliably provide a full picture of coverage).

Describe the solution you'd like

We have a couple of different paths we could take

• Make Carry Forward Flags more performant
  • This is not the most appealing option, as it forces the normalizing of creation of flags at a per - test level and requires investment to support that paradigm (more uploads per commit etc).
• Continue the work scoped out for carrying forward test results (Label Analysis) within bazel setups https://github.com/codecov/engineering-team/issues/106

[vlad-ko]

I would like to propose another option, by decoupling the carry-forward functionality from the flags feature (reporting). Let's call it CFRs (carry forward reports).

This is brainstorming, but hopefully it will offer something for further investigation:

cfr_management:
  default_rules: # the rules that will be followed for any CFR
    carryforward: true
    branch: dev
  individual_cfrs: # exceptions to the default rules above, stated for individual reports (optional)
    - name: my_report  #fill in your own flag name
    - ...

To create a CFR in the system we could create a CLI option create-cfr my-report to the do-upload command, for example. To stop the report from propagating we could run remove-cfr my-report.

Overall the goal is to say, we'd like to carry-forward this report on a given branch or otherwise. And have some option to disable this functionality if the test suite is removed or renamed.

[rohan-at-sentry]

More context dumping and outlining our options here https://www.notion.so/sentry/Better-Supporting-Customers-that-run-Test-Selection-on-Every-Commit-d30c6ad0e87240938dcdbec85acc5c07?pvs=4