[FEATURE] OpenSearch Project Distribution Build and Integration Test Analytics

prudhvigodithi commented 1 month ago

Is your feature request related to a problem?

Coming from the comment https://github.com/opensearch-project/opensearch-build/issues/4171#issuecomment-2147976422, ingest the OpenSearch and Dashboard plugins distribution and Integration test build results. Similar to what we have today with OpenSearch Gradle Check Metrics ingesting the build and Integration test results should give us lot of insights and data regarding the build failures data for a given release and this would give us trends over the period. This is also one of the main requirement for Central Release Dashboard as proposed here.

Having this distribution and Integration test build results in metrics cluster should also the release management for the release manager where the visualizations links can be shared directly instead of parsing through build jobs and logs and create a complex markdown table for failures for each plugin, distribution, architecture etc. Example from past release https://github.com/opensearch-project/opensearch-build/issues/3616#issuecomment-1636583248, https://github.com/opensearch-project/opensearch-build/issues/3616#issuecomment-1638704742

What solution would you like?

Create a Jenkins library and use it in post section that can collect the build data in the following format. Having a library gives us would give us advantage of synchronous data ingestion to the metrics cluster which can collect and push the data at the end of the build.

Distribution Build

{
  component: alerting
  component_repo_url: https://github.com/opensearch-project/alerting
  version: 2.16.0
  ref: abcgdtxgrsggvgfcfds
  distribution_build_number: 10020
  distribution_build_url: https://build.ci.opensearch.org/job/distribution-build-opensearch/10020/display/redirect
  build_start_time: <JENKINS_BUILD_TIME>
  component_build_result: failed
  rc: false
  rc_number: 0
}

Integration Test Build

{
  component: securityDashboards
  repo_url: https://github.com/opensearch-project/security-dashboards-plugin
  version: 1.3.18
  build_number: 6040
  build_url: https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/6040/display/redirect
  distribution_build_number: 7791
  distribution_build_url: https://build.ci.opensearch.org/job/distribution-build-opensearch-dashboards/7791/display/redirect
  build_start_time: <JENKINS_BUILD_TIME>
  component_build_result: failed
  rc: false
  rc_number: 2
  platform: windows
  architecture: x64
  distribution: zip
  test_report_manifest: https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.18/7791/windows/x64/zip/test-results/6040/integ-test/test-report.yml
  with-security: failed
  with-security_build_yaml: https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.18/7791/windows/x64/zip/test-results/6040/integ-test/securityDashboards/with-security/securityDashboards.yml
  with-security_cluster_stdout: [https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.18/7791/windows/x64/zip/test-results/6040/integ-test/securityDashboards/with-security/local-cluster-logs/id-0/stdout.txt, https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.18/7791/windows/x64/zip/test-results/6040/integ-test/securityDashboards/with-security/local-cluster-logs/id-1/stdout.txt]
  with-security_cluster_stderr: [https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.18/7791/windows/x64/zip/test-results/6040/integ-test/securityDashboards/with-security/local-cluster-logs/id-0/stderr.txt, https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.18/7791/windows/x64/zip/test-results/6040/integ-test/securityDashboards/with-security/local-cluster-logs/id-1/stderr.txt]
  without-security: failed
  without-security_build_yaml: https://ci.opensearch.org/ci/dbc/integ-test/1.3.18/10042/windows/x64/zip/test-results/8364/integ-test/k-NN/without-security/k-NN.yml
  without-security_cluster_stdout: [https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.18/7791/windows/x64/zip/test-results/6040/integ-test/securityDashboards/without-security/local-cluster-logs/id-2/stdout.txt, https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.18/7791/windows/x64/zip/test-results/6040/integ-test/securityDashboards/without-security/local-cluster-logs/id-3/stdout.txt]
  without-security_cluster_stderr: [https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.18/7791/windows/x64/zip/test-results/6040/integ-test/securityDashboards/without-security/local-cluster-logs/id-2/stderr.txt, https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.18/7791/windows/x64/zip/test-results/6040/integ-test/securityDashboards/without-security/local-cluster-logs/id-3/stderr.txt]
}

Once the data is collected the library should parse and index the data to opensearch-integration-test-results-${formattedDate}, opensearch-distribution-build-results-${formattedDate}.
Create useful visualizations like data table that can be filtered by release, component, rc, with-security, without-security etc and get the log details from the links along with other information about the number of rc's.
Having this data over the period should give trends (using time series graphs) about the release and mainly with failing builds and tests for a given release.

prudhvigodithi commented 1 month ago

Adding @getsaurabh02 @peterzhuamazon @reta @dblock to please take a look. Thank you

peterzhuamazon commented 1 month ago

We can reduce the url length but it is a good start 👍 .

Will use these as part of the Central Release Dashboards here: #51 .

Thanks.

reta commented 1 month ago

Certainly +1, thanks a lot @prudhvigodithi !

prudhvigodithi commented 1 month ago

Able to collect the data and index to my test cluster, will create the library PR's shortly.

prudhvigodithi commented 1 month ago

We can reduce the url length but it is a good start 👍 .

Yes Peter, we should either use some open source tiny URL framework or we can leverage dashboard scripted fields to do this, for every URL fields have an associated scripted field for the short URL. Thanks

prudhvigodithi commented 1 month ago

Created visualizations that contain all the build and integration test information. Please check them at OpenSearch Metrics Dashboard&_a=(description:%27OpenSearch%20Release%20Build%20and%20Integration%20Test%20Results%27,filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),query:(language:kuery,query:%27%27),timeRestore:!t,title:%27OpenSearch%20Release%20Build%20and%20Integration%20Test%20Results%27,viewMode:view)). With this setup, we can easily filter by build (RC build number) to get all the build and integration test details for both with-security and without-security configurations, including URLs to download the test reports. There are many other filters available, allowing us to slice and dice the data by architecture, distribution, platform, OS, or OSD components.

prudhvigodithi commented 3 weeks ago

Closing this issue as the metrics dashboard now has the OpenSearch Project Distribution Build and Integration Test Analytics. Readme PR https://github.com/opensearch-project/opensearch-metrics/pull/60. This dashboard is used for 2.16.0 to track the build and integration test failures per RC and post the release updates, example https://github.com/opensearch-project/opensearch-build/issues/4771#issuecomment-2273857626. Thanks @getsaurabh02

opensearch-project / opensearch-metrics