andrross
commented
4 months ago The request here is very similar to this older issue: #3713
Closed andrross closed 1 day ago
andrross
commented
4 months ago The request here is very similar to this older issue: #3713
peternied
commented
2 months ago @andrross I created a repo [1] that collects project health information and published reports to its repo ever day - see the latest reports at https://github.com/peternied/contribution-rate?tab=readme-ov-file#reports
One such report is a last 30 days top failing test - here is the March 8 report. It will keep updating every day. Feel free to contribute any kind of reports you'd like to see.
prudhvigodithi
commented
3 weeks ago Hey @andrross @peternied we now have the gradle metrics published to OpenSearch Gradle Check Metrics dashboard, this is part of surfacing opensearch-metrics to community. Please check the current supported metrics. How about we expand this and add more metrics as required? and also we use this data for creating triggers like GitHub issues, comments etc. @bbarani @Pallavi-AWS @dblock
andrross
commented
2 weeks ago @prudhvigodithi Do you think it would make sense to add some details about using the gradle check metrics dashboard to help investigate and fix flaky failures in either TESTING.md or DEVELOPER_GUIDE.md?
@dreamer-89 created a great list in #3713:
I think if we document somewhere how to use the new dashboard to solve those problems then we can close both of these issues as completed.
prudhvigodithi
commented
2 days ago Thanks @andrross and @dreamer-89, based on the list you have I have modified the gradle check workflow with new fields and created some new visualizations based on the indexed data (Thanks to @rishabh6788 for setting up the initial flow), please check the link OpenSearch Gradle Check Metrics.
For this I have created a pie chart with the top test_class that has the majority of the failures, this chart should also have the top failing tests within this test_class, we can further slice and dice the data for getting the list of PR's and owners or with post merge, that has top failing tests upon filter.
The following data tables should have the git commit, the associated PR and the PR owner with all the failing test details, we can filter per PR or commit to get the details of failed tests. The new visualization Gradle Check - Top test class failures with Post Merge also has the flaky test information, its associated commitID and PR (with owner) that was merged with this commitID with post merge (gradle check that ran after the PR is merged) action. We should be able to further drill down with test name or the test class name for more details.
For this the dashboard has a TSVB and line chart with the trend for the failure tests, this can be again further filtered with test name, test class, commitID, PR and with executions with Post merge.
The entire visualizations can be filtered with PR owner, PR number or commitID. The results has the hyperlinks for the GitHub PR or commit where one can see the comments and other users. The dashboards also has the PR owner attached to see impacted user. The visualizations also has the hyperlinks with the jenkins build data where one can see all the stack trace details for the failed tests (example 39487).
All the visualizations in dashboard can be filtered by date range, using OpenSearch we get this out of the box :) With this we can go back and see the trends and infer results based on it.
Adding @peternied @getsaurabh02 @dblock @Pallavi-aws @reta
reta
commented
1 day ago @prudhvigodithi @rishabh6788 it looks great, thank you so much folks for putting it all together
andrross
commented
1 day ago @prudhvigodithi @rishabh6788 it looks great, thank you so much folks for putting it all together
Agreed, this is awesome!
prudhvigodithi
commented
1 day ago Thanks @reta and @andrross, I have a PR created with some details added to the DEVELOPER_GUIDE.md regarding this dashboard https://github.com/opensearch-project/OpenSearch/pull/13919, please check.
prudhvigodithi
commented
1 day ago Next step moving forward for surfacing the test failures as GitHub Issues instead of creating a very generic issue like https://github.com/opensearch-project/OpenSearch/issues/13893 (coming from https://github.com/opensearch-project/OpenSearch/blob/main/.github/workflows/gradle-check.yml#L161-L168) which sometimes fails to execute https://github.com/opensearch-project/OpenSearch/actions/runs/9320653340/job/25657907035, how about we use the following data table information to create a GitHub issue.
We should start by creating an issue at a test class level NestedQueryBuilderTests, link and keep updating all the commits and PR information to the issue created for NestedQueryBuilderTests.
1st to the issue created for NestedQueryBuilderTests, we can link all the post merge failures and commits.
2nd on the same issue for NestedQueryBuilderTests, we can add the failed tests which are part of NestedQueryBuilderTests and Jenkins build information for stacktrace.
3rd on the same issue, we can add other PR's information where this has or has been failing.
I'm open for ideas on whom to assign this created issue? Should we just keep it open without any assignee as each issue will have multiple PR and commits information. later during triaging the maintainer should be able to identify the right team/user and add as assignee.
Moving forward we can have a logic to auto close the created issue if in last 30 days there is no failure for the test class (NestedQueryBuilderTests in above example) found in post merge Gradle Check build and reopen as required.
@andrross @reta @dblock @getsaurabh02 @peternied let me know your thoughts on this.
Thank you
andrross
commented
1 day ago Now that we have the metrics and the updated developer guide, I'm going to close this and issue #3713. If anyone thinks there is more to do here please reopen or open a new issue. Thanks!
I've created a script that crawls the OpenSearch Jenkins builds to find test failures, but only for the Gradle checks that run on code after it is pushed to the main branch. This filters out failures that are due to unmerged code in work-in-progress PRs.
I've included below the output after crawling 2000 recent builds (approx. Oct 16 - Nov 14). This data is very hard to follow, but one thing in particular stands out:
SearchQueryIT.testCommonTermsQueryis a frequently failing test, but only since build 29184 (Oct 28). There are no failures before that, which strongly suggests something was changed around Oct 28 that introduced the flakiness. ~I haven't started to look but I suspect we'll be able to find the cause pretty quickly given that there is a point in time to start looking at.~ Update Nov 16: the root cause was an unrelated change for concurrent search randomly increased the number of deleted documents and exposed some underlying brittleness in this test: #11233 Diagnosing the root cause was a bit tricky and required diving into the specifics of how the common terms query works, but it was indeed much simpler once the flakiness was correlated to a small date range and then a specific commit.Surely there are better tools for visualizing test reports over time, perhaps already built into Jenkins? Also, we don't push that many commits so the sample size on builds after pushes to main isn't that large. Something like a nightly job to run the test suite 10 or 50 or 100 times and create a report on failures would help to quickly surface newly introduced flakiness.