search

opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
262 stars 201 forks source link

Refactor data prepper release process #2122

Closed gaiksaya closed 1 year ago

gaiksaya commented 1 year ago

Is your feature request related to a problem? Please describe. There has been a development in the way artifacts are released. We would like to re-do the data-prepper release process to use the new process https://github.com/opensearch-project/opensearch-build/issues/1234

Old release process jenkins file : https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/data-prepper/release-data-prepper-all-artifacts.jenkinsfile

Describe the solution you'd like

  1. Add release-drafter GitHub action workflow that will build the artifacts and upload the artifacts to a draft release. Sample workflow
  2. Add a new jenkins file to this repository that downloads those artifacts from the above draft release and uses a new lib called publishToMaven that autopublishes the maven artifacts to maven. Similary, other libraries such as copyContainer for copying container images to different platforms. This jenkins File uses draft release as a trigger.

In this new process, all that a maintainer needs to do is push a tag and the releasing artifacts to different platforms will be taken care by the GHA and jenkins workflows.

Describe alternatives you've considered (Optional) Continue to use what we have.

Additional context Add any other context or screenshots about the feature request here.

Design

See comment: https://github.com/opensearch-project/data-prepper/issues/2122#issuecomment-1662414428

Tasks

dlvenable commented 1 year ago

@gaiksaya , I believe we are almost doing this process.

We have a release GHA which builds artifacts and uploads them to our staging repository. This is similar to the release-drafter requested.

We also have an existing Jenkinsfile which you linked to. This copies from the artifacts we have above.

It sounds like the gap we have is only related to triggering these jobs. I understand you are looking for the following changes.

  1. Trigger the release job when a tag is pushed
  2. Trigger the Data Prepper Jenkinsfile when the release build is complete.

One thing that we do want to account for here is an idea of nightly releases. We've had several requests to provide nightly or weekly builds. We'd need to be sure that for such release builds, they do not trigger the Jenkinsfile.

gaiksaya commented 1 year ago

Yes, trigger when tag is cut. The current process needs a manual run by providing input the parameters. Trying to see how can we get out of that. Few changes than we have made since then include maven publishing being completed automated. No manual intervention is required now (Need to make few changes in that jenkins file)

Regarding nightly builds, what artifacts would that be? I see multiple artifacts in this process.

bbarani commented 1 year ago

@dlvenable Can you please let us know the status of on-boarding Data prepper to one click release process? We do not have bandwidth to support manual releases and I don't want that to become a blocker for any urgent releases.

dlvenable commented 1 year ago

@bbarani , We are working on this for the 2.3 release and plan to start using it with that release.

dlvenable commented 1 year ago

Overall Approach

The overall approach is to trigger the existing Data Prepper Jenkins job using a custom draft release trigger on Jenkins. This will be integrated into the existing Data Prepper release GitHub Action job.

You can conceptually view the release as:

  1. Build artifacts (existing)
  2. Smoke tests (existing)
  3. Confirm release using a GitHub issue.
  4. Start the Jenkins promotion job.

Release GitHub Action

We will add the following steps to the existing Data Prepper GitHub Action:

  1. Create a release issue which requires approval from two Data Prepper maintainers.
  2. Generate a release descriptor YAML file. See details below.
  3. Draft a GitHub release. Attach the release descriptor YAML to this draft release.

The Data Prepper repository will have a custom trigger for draft releases to start a Jenkins job. Thus, step 3 above will trigger Jenkins.

Jenkins Trigger

The opensearch-build project will need a new Jenkins trigger for Data Prepper. This trigger will be somewhat different than the existing trigger used by clients.

This trigger will:

  1. Get the draft release.
  2. Download the release descriptor from the draft release.
  3. Start the existing Jenkins job for promoting Data Prepper releases.

Release descriptor format.

The release descriptor file will be a fairly straightforward YAML file.

A simple example which relates to our 2.3.2 release build (build 33 on Jenkins).

version: 2.3.2
build_number: 60
release_major_tag: true
release_latest_tag: true
asifsmohammed commented 1 year ago

Create a release issue which requires approval from two Data Prepper maintainers. How can we get approvals on an issue?

I thought the GA is going to create a PR for release.

gaiksaya commented 1 year ago

Sample issue @asifsmohammed : https://github.com/opensearch-project/opensearch-py/issues/462 Uses this GH action: https://github.com/opensearch-project/opensearch-py/blob/main/.github/workflows/release-drafter.yml#L17-L24

dlvenable commented 1 year ago

@asifsmohammed , It is not a PR, but a GH issue as @gaiksaya noted.

The GHA will poll this issue looking for two comments from maintainers which have the known text.

You can see an example in my current testing right here: https://github.com/dlvenable/data-prepper/issues/316