Adal3n3
commented
4 months ago Here is the user testing feedback and team discussion: Notion link
Discussion note: (with @rohan-at-sentry @joseph-sentry)
- PR comment: The feedback is “1/overwhelming test details… 2/developers might not go thru these details and 3/ no one wants to sit here and just wait for the PR” should we consider remove or do something for the two dropdowns for failed and flaky tests details? But then we won’t be able to identify which test is flaky for them.
- ok for now but maybe show test name only in the future, and/or can make the tests details configurable, choose to turn on/off. Here’s the on-going supporting works:
- https://github.com/codecov/engineering-team/issues/1487
- https://github.com/codecov/engineering-team/issues/1233
- https://github.com/codecov/engineering-team/issues/1551
- PR comment: Should we add a summary table for the number of tests, duration for all tests run and failure rate so developer can get a summary view before dig into details.
- Total failed tests run: we can get the duration from the check run, might not be useful bc usually it doesn’t have a big change, and non-actionable piece of info so should not show on the PR.
- Failure rate: same as above, so no failure rate on the PR.
- PR comment: Should we remove the line “To report false or accurate flaky tests detection, please use the 👍 or 👎 icon.” because we heard 1/ customers may not sit and wait for PR, 2/ may not give us the feedback we want.
- We will remove the feedback loop in PR
- We don’t need the Mute and mark as flaky functionalities in the UI for now.
- Should we explore tests for coverage? Separate discussion
- Should we prioritize provide tests analytic API?
- We want to do this but not sure on the timing. Maybe when we know if we want to GA
- Shorten the test name?
- https://github.com/codecov/engineering-team/issues/1551
- Start by default which is the very long name, but can make it configurable in the future.
- Failure rate is available in Q2
- For the tests summary dashboard, we have all the data available, the question is timing on getting these data to Application team. Maybe Application team can use mock data for building the data table, if timing doesn’t align. - Aj to review the dashboard for dev-ready.
The challenge is that developers often investing significant time identifying flaky tests within their failed tests result, which can be a frustrating process as these tests are meaningless and contribute to CI failures. By identifying flaky tests in the pull request comments for customers, they can quickly investigate the cause of the failure and track the history of the flakes, leading to faster addressing of flaky tests.
Describe the solution you'd like Private beta design:
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context Add any other context or screenshots about the feature request here.
Figma: link