Premature Email Notifications from Codecov on Parallel Coverage Reports

[codecovdesign]

Problem Description

When running tests in parallel across multiple workers, our project sends several coverage files to Codecov. The current behavior of Codecov triggers an email notification upon processing the first coverage file, often leading to premature and sometimes inaccurate coverage information.

Solution ideation

Explore

• how to better show after_n_build; consider it depends on n of uploads
• related: https://github.com/codecov/feedback/discussions/204
• in cases that after_n_build is applied, there is the issue where a report may not show since n has not been satisfied (sometimes due to carryforward) - how could the logic be fixed to show report and/or user awareness why report isn't showing. example: https://github.com/getsentry/sentry/pull/66447

[giovanni-guidini]

We already have a better approach for this. The emails come from GitHub when we post a comment to a PR, and I don't think we can prevent them from doing so.

We have a "manual trigger" for notifications that can help with parallel processing though. First you set the codecov.notify.manual_trigger to true in your codecov.yml Then after your CI has completed and all uploads were made you issue a codecovcli send-notifications command to let codecov know the uploads are complete and you can send notifications.

We can probably look into adding this to the codecov action so users don't have to install the CLI to use that feature