codecov / engineering-team

This is a general repo to use with GH Projects
1 stars 1 forks source link

Flaky Tests Reporting should reliably work well for 100 JS / Python Customers #1188

Closed joseph-sentry closed 2 months ago

joseph-sentry commented 7 months ago
### Onboarding
- [ ] https://github.com/codecov/engineering-team/issues/1229
### Milestone 0
- [x] Adjust database models to accomodate milestone 0 goals
- [x] Match failures with existing flakes on new test results
- [x] PR comment notifies users of failures that occured for tests that are flaky on main
- [x] Process flakes into DB on new test results from main
- [x] Process flakes into DB on merge of test results from PR into main
- [x] Expire flakes after 30 consecutive passes of that test
- [ ] https://github.com/codecov/engineering-team/issues/2032
- [ ] https://github.com/codecov/engineering-team/issues/2033
### Ingestion improvement
- [ ] https://github.com/codecov/engineering-team/issues/1486
- [ ] https://github.com/codecov/engineering-team/issues/1750
### Display
- [ ] https://github.com/codecov/engineering-team/issues/1232
- [ ] https://github.com/codecov/engineering-team/issues/1296
- [ ] https://github.com/codecov/engineering-team/issues/1384
- [ ] https://github.com/codecov/engineering-team/issues/1704
### Measuring success
- [ ] https://github.com/codecov/engineering-team/issues/1604
- [ ] https://github.com/codecov/engineering-team/issues/1688
### For future consideration
- [ ] https://github.com/codecov/engineering-team/issues/1487
- [ ] Summarize error messages (requires error message parsing)
- [ ] Link to source code from comment (requires error message parsing)
- [ ] Improvements to notifications synchronization, unifying pipeline
rohan-at-sentry commented 7 months ago

@joseph-sentry for your onboarding step can you help us understand

  1. What is the flow you're thinking of where setup-codecov action is beneficial? Does that now mean we have 2 actions?
  2. Integrating existing Codecov action with test result ingestion - does invalidate Add option for uploading both report types at once (codecov-cli do-upload --test-results) for most users (at least on GH actions and orb if we do that)
joseph-sentry commented 7 months ago

For the onboarding I think we should have a setup-codecov action no matter what, so that users have an easier way to install the CLI without using pip. I think it would be beneficial in general for the CLI not just for test result ingestion. But I think a setup-codecov action would make it easier to use test result ingestion, since right now, we require users to use the CLI to upload their test results.

I think a setup-codecov action removes the need for the later two tasks in Onboarding, those are more of a want at that point

rohan-at-sentry commented 6 months ago

@trent-codecov - the KR here is to deliver the supporting pieces to enable 100 customers to use a flaky test reporting offering by end of Q2.

the big pieces are

The task list should now encapsulate the shortest path to fulfilling this KR