[Meta] OpenSearch Release Improvements

Coming from https://github.com/opensearch-project/opensearch-build/issues/3676 A Central release workflow to support OpenSearch public release and https://github.com/opensearch-project/opensearch-build/issues/2080, following are the issues that can help improve the OpenSearch Release process. Once the following issues are achieved it would be easy to achieve the long term goal of one-click release or having an automated release tool that can release the OpenSearch bundle.

Please consider the following suggestions as areas for improvement in the OpenSearch release, aiming to enhance the efficiency and reduce the time and effort required for a release manager.

For more details about the OpenSearch release process, refer the release process guide RELEASE_PROCESS_OPENSEARCH.md

Enhancements

[x] Circuit Breaker for AUTOCUT issues
There are lot of AUTOCUT issues created and closed because of flaky tests, have a mechanism to reduce the number of AUTOCUT issues created. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/4171
[x] Central workflow to execute pre-release promotion jobs. Related issue: https://github.com/opensearch-project/opensearch-build/issues/4136 To have one single workflow to execute all the pre-release and all promotion jobs.
[x] Proper Labelling for AUTOCUT issues. Have proper labels added to the created AUTOCUT issues, example like if the failure is per distribution add the distribution label to the created issues. Later it would be easy to filter the issues per label.
[x] Consolidated Release notes automation The process to gather the consolidated release notes and raise a PR is now done manually by a release manager, need an automation to generate and raise a PR. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/4009
[ ] Jenkins upgrade Cycle. Upgrade the Jenkins, come up with a quarterly upgrade path based on released LTS version Related Issue: https://github.com/opensearch-project/opensearch-ci/issues/333
[x] Split Public Jenkins into multiple masters. Related Issue: https://github.com/opensearch-project/opensearch-ci/issues/373
[x] Incremental builds Currently we are building all components sequentially and we have to rebuild all components to accommodate even small commit changes. This incremental build process will allow to rebuild only specific updated artifacts and save the unnecessary time on rebuilding other unchanged plugins. It call reduce the distribution bundle generating time from 1 hour to 10 minitues ideally. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/3922
[x] Refactor usage of existing index.json used in the latest URL. We plan to store the latest of each distribution type separately in different index.json files and retrieve the latest from the corresponding index.json that is applicable based on the distribution type specified in the url. Related issue: https://github.com/opensearch-project/opensearch-build/issues/4151
[ ] OpenSearch Build repo versioning framework (Resolves the need of branching). Related Issue: https://github.com/opensearch-project/opensearch-build/issues/2490
[x] OpenSearch Playground with daily build artifact. Related Issue: https://github.com/opensearch-project/opensearch-devops/issues/129
[x] Integrate Validation mechanism with every OS and OSD build. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/3441
[x] Mechanism to auto create the release branch for the plugin repos. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/4005
[x] Introduce placeholder in manifests to allow skip the component or use the placeholder component for any other use-case. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/3707
[ ] Open source the signing mechanism Related Issues: https://github.com/opensearch-project/opensearch-build/issues/118, https://github.com/opensearch-project/opensearch-build/issues/3468
[ ] Once the RC is finalized, we run several tests on demand, having a central workflow that takes input of the RC numbers, runs all the test jobs sequentially and post the results in different mediums. This should include the benchmark tests as well.
[x] One central workflow to trigger all promotion job for releasing. Have a fall back mechanisms in place.
[ ] Option to choose types of promotion in case of any job failure. To is to make sure the promotion jobs are only incremental from failure.
[ ] CI docker images for build and test should be maintained per version. https://github.com/opensearch-project/opensearch-build/issues/4233
[x] Add Cloudfront Invalidations https://github.com/opensearch-project/opensearch-build/issues/4262
[ ] Add rollback after artifacts promoted to the prod bucket https://github.com/opensearch-project/opensearch-build/issues/4263

Bugs

[ ] Merge the Jenkins Create/Close Github libraries to central library Today we have multiple libraries for creating and closing the github issue which is adding some confusion and more effort to manage the libraries. Need to merge the logic into one library and use it for all Github actions. closeBuildSuccessGithubIssue closeGithubIssue createBuildFailureGithubIssue createGithubIssue
[x] S3 Upload fails randomly Related Issues: https://github.com/opensearch-project/opensearch-build/issues/3097 https://github.com/opensearch-project/opensearch-build-libraries/issues/319
[ ] New ci-runner user cause weird failures in integ-tests Related Issues: https://github.com/opensearch-project/opensearch-build/issues/4241
[ ] Version Increment bug with OSD. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/3856
[ ] Validate POM Files before pushing to maven repo (nexus). Related Issue: https://github.com/opensearch-project/opensearch-build/issues/2928
[ ] Distribution build failure AUTOCUT issues do not provide detail information Related Issue: https://github.com/opensearch-project/opensearch-build/issues/4155
[ ] Smaller Docker Images of OpenSearch and OpenSearch Dashboards. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/3853
[ ] RPM/DEB permissions update. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/3815
[ ] Move the logic that is used for build which is now part of groovy libs to python. Move the validation groovy code to the validation python workflow. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/2609
[ ] AUTOCUT issues does not honor windows failure https://github.com/opensearch-project/opensearch-build/issues/4314

Documentation

[ ] GitHub Wiki for OpenSearch Build repo. Related Issue: https://github.com/opensearch-project/opensearch-build/issues/3936
[ ] Release templates As a release manager every time one has to type and update the release issue, it would be easy if we have any templates (like RC is ready, Integ tests failing, starting the release process, pending version increment PR’s etc) to use them to update the release issue. An automation later that should auto tag the repo owners/maintainers.

opensearch-project / opensearch-build