Crate a short plan that describes the how the "dream" release process should be

[jvanz]

Research and describe a possible solution to automate the process of releasing Kubewarden components.

[jvanz]

The Kubewarden release process need to be splitted in 2 scenarios. As described in the Kubewarden versioning RFC, all the main components of the stack share the same major and minor versions. But the patch version is independent. Therefore, the release process of patch update can run independent of other components.

To quick off the discussion, let me share a suggestion of a flow to automatically release patch updates. The following flow uses the controller as example, but it should be reproducible in all main components:

graph TD
A[Tag patch version of kubewarden-controller] 
B[CI and tests]
C{Tests passed}
D[Run e2e tests with the latest changes]
E{E2E tests passed}
F[Trigger kubewarden-controller chart update]
G[Update kubewarden-controller chart patch version]
H[Release kubewarden-controller chart]
I[Create kubewarder-controller github release]
J[Notify kubewarden developers]
K[Delete tag which triggered the job]
L[Abort release]
M[Update changelog]

style A fill:#bfb,color:black
style B fill:#bfb,color:black
style C fill:#bfb,color:black
style D fill:#bfb,color:black
style E fill:#bfb,color:black
style M fill:#bfb,color:black
style I fill:#bfb,color:black
style F fill:#bfb,color:black
style J fill:#bfb,color:black
style K fill:#bfb,color:black
style L fill:#bfb,color:black

style G fill:#bbf,color:black
style H fill:#bbf,color:black

A --> B
B --> C
C -->|Yes|D
C -->|No| J
D --> E
E -->|Yes| M
M --> I
E -->|No| J
I --> F
F --> G
G --> H
J --> K
K --> L

The goal of this flow is to allow maintainers just tag a new tag updating the patch version of the component and all the release process happen automatically from that. Considering the above flowchart, all the steps represented in green boxes happen in the component repository CI (in this example the kubewarden-controller). The blue one in the helm-charts repository.

I think we can use the updatecli to update helm charts versions. As well as github actions trigger to fire CI/CD job in another repositories from the components repositories. We can try to use the same tool used by Epinio to trigger the jobs in another repositories.

[jvanz]

I'm moving this issue to "Pending review" because I would like to get some initial feedback from the @kubewarden/kubewarden-developers

[raulcabello]

Overall flow looks good. A few considerations:

• I'm guessing update changelog happens automatically, right? Are we going to keep them in git?
• Currently e2e tests takes more than 30 minutes, and it will increase as we add more tests. Anyway I think it makes sense to run them. Just keep in mind it will take time to do a patch release.
• Update kubewarden-controller chart patch version This should happen automatically. How are we going to do this? Are we going to commit to main directly the changes needed? Create a PR? can updatecli do this?
• Release kubewarden-controller chart For this step are we just going to create a tag in the helm-charts repo from the gha that would trigger a release? If so, we need the changes from the previous step already in main

[viccuad]

Looks good to me too. About the changelog, I would collapse Update changelog and Create kubewarder-controller github release into 1, and the GH release contains the generated changelog.

I'm not picky with the workflow, I think it's a good starting point, and we don't need a super defined workflow, we can iterate. I really look forward to bumping helm-chart versions with updatecli, and creating changelogs on the fly.

[jvanz]

I'm guessing update changelog happens automatically, right? Are we going to keep them in git?

Yes, it should be done automatically. I would say to add it in the GH release. But if we need to add it in the git, it should be updated automatically by the CI job.

• Update kubewarden-controller chart patch version This should happen automatically. How are we going to do this? Are we going to commit to main directly the changes needed? Create a PR? can updatecli do this?

Yes, I'm thinking about using updatecli to do this for us. I would commit it directly into the repository. Otherwise, we need another human intervention into the release process (e.g. approving the PR). But we can debate this if you think is dangerous.

Release kubewarden-controller chart For this step are we just going to create a tag in the helm-charts repo from the gha that would trigger a release? If so, we need the changes from the previous step already in main

I'm thinking to use the current approach. Once the version defined in the Chart.yaml file is updated the release job will run the chart-releaser and it will create the tags for us.

[jvanz]

About the changelog, I would collapse Update changelog and Create kubewarder-controller github release into 1, and the GH release contains the generated changelog.

Sure thing! I've split it into a separate task just to make clear in the workflow that it will be automatically done by the CI/CD.

[jvanz]

In the major and minor updates we need to keep kwctl, kubewarden-controller and policy-server versions in sync. In other words, all of these components should have the same major/minor number.

For this scenario, the component release phase is pretty similar to the patch version. Actually, I believe we can use the same workflow for both purposes. The difference will be what is done in the helm chart workflow. If the update is triggered by a patch release, just release the correspondent chart of that component. If it is a major/minor release, all the 3 main component should updated.

flowchart TB
subgraph release component version
A[Tag patch version of kubewarden-controller] 
B[CI and tests]
C{Tests passed}
I[Create kubewarder-controller github release]
J[Notify kubewarden developers]
L[Abort release]
M[Update changelog]
N[Trigger Kubewarden chart release]

style A fill:#bfb,color:black
style B fill:#bfb,color:black
style C fill:#bfb,color:black
style M fill:#bfb,color:black
style I fill:#bfb,color:black
style J fill:#bfb,color:black
style N fill:#bfb,color:black
style L fill:#bfb,color:black

A --> B
B --> C
C -->|Yes| M
C -->|No| J
M --> I
J --> L
I --> N
end

subgraph release kubewarden charts 

s3{Is patch release?}
s4{All the components has the same released version?}
s5{Is policy-server release?}

s6[Update kubewarden-defaults chart version]
s8[Update kubewarden-controller chart version]
s14[Update kubewarden-crds chart version]
s7[Release kubewarden-defaults chart]
s9[Release kubewarden-controller chart]
s15[Release kubewarden-crds chart]
s23[Run E2E tests with the latest changes]
s24{E2E tests passed?}

s10[Update kubewarden-defaults chart version]
s12[Update kubewarden-controller chart version]
s16[Update kubewarden-crds chart version]
s21[Run E2E tests with the latest changes]
s22{E2E tests passed?}
s11[Release kubewarden-defaults chart]
s13[Release kubewarden-controller chart]
s17[Release kubewarden-crds chart]

s18[Finish]

s19[Run E2E tests with the latest changes]
s20{E2E tests passed?}

s25[Abort]

style s3 fill:#bbf,color:black
style s4 fill:#bbf,color:black
style s5 fill:#bbf,color:black
style s6 fill:#bbf,color:black
style s7 fill:#bbf,color:black
style s8 fill:#bbf,color:black
style s9 fill:#bbf,color:black
style s10 fill:#bbf,color:black
style s11 fill:#bbf,color:black
style s12 fill:#bbf,color:black
style s13 fill:#bbf,color:black
style s14 fill:#bbf,color:black
style s15 fill:#bbf,color:black
style s16 fill:#bbf,color:black
style s17 fill:#bbf,color:black
style s18 fill:#bbf,color:black
style s19 fill:#bbf,color:black
style s20 fill:#bbf,color:black
style s21 fill:#bbf,color:black
style s22 fill:#bbf,color:black
style s23 fill:#bbf,color:black
style s24 fill:#bbf,color:black
style s25 fill:#bbf,color:black

s3 -->|No| s4
s4 -->|No| s18

s3 -->|Yes| s5
s5 -->|Yes| s6
s6 --> s19
s19 --> s20
s20 -->|Yes| s7
s5 -->|No| s8
s8 --> s14
s14 --> s23
s23 --> s24
s24 -->|Yes| s9
s9 --> s15

s4 -->|Yes| s10
s10 --> s12
s12 --> s16
s16 --> s21
s21 --> s22
s22 -->|Yes| s11
s11 --> s13
s13 --> s17

s20 -->|No| s25
s22 -->|No| s25
s24 -->|No| s25

end

N -..-> s3

[jvanz]

@kubewarden/kubewarden-developers , I've decided to not add in the proposed steps for the major/minor release process a step to automatically tag other components just to avoid the pain of deciding how to choose which commit should be consider ready for release. But we can discuss that if you think is worth it for this first iteration.

[kkaempf]

I'm missing a visible "sync" step in the flowchart which ensures "... kwctl, kubewarden-controller and policy-server versions in sync."

[viccuad]

I'm missing a visible "sync" step in the flowchart which ensures "... kwctl, kubewarden-controller and policy-server versions in sync."

Maybe an intermediate state "trigger sync release of kubewarden-controller & policy-server"? (in the case of a kwctl release for example)

Apart from that, looks good. I fear thinking too much about it.

[flavio]

I'm fine with the proposal. It's good to have a plan about the release process of all the components, I would start with the automation of the patch releases and then, after some 1st hand experience, implement the bigger plan.

I've just once concern about the patch level automation: the requirement to have a successful run of e2e tests after a tag event. The e2e tests take quite some to run and can fail because of false positives. How can we recover from such a failure (like an innocent timeout)? I would also expect e2e tests to be run before the tag is created. This could be done manually, we could discuss on a per-case basis whether it makes sense to run the e2e tests or not. Long story short: I would just remove the e2e tests for now. But this is just my personal opinion

[jvanz]

We had a quick chat to discuss the proposed solution. We decided to, after a small change, move forward and implement it. The only change is the removal of the E2E tests as a gate to decide if a release will continue or not. The team is afraid that the E2E could fail sometimes due timeout issues and other factors that is not an actual issue in the Kubewarden components. Therefore, we have decided to run the E2E tests in parallel of the usual release process to spot possible issues as fast as possible. But the tests will not be able to abort the release.

[jvanz]

I'm missing a visible "sync" step in the flowchart which ensures "... kwctl, kubewarden-controller and policy-server versions in sync."

The step "All the components has the same released version?" should be the step waiting for all components (policy-server, kubewarden-controller and kwctl) sync their releases.

[jvanz]

The "final" version of this first iteration for the release workflow would be:

flowchart TB
subgraph Release component version
A[Tag patch version of kubewarden-controller] 
B[CI and tests]
C{Tests passed}
I[Create kubewarder-controller github release]
J[Notify kubewarden developers]
L[Abort release]
M[Update changelog]
N[Trigger Kubewarden chart release]

style A fill:#bfb,color:black
style B fill:#bfb,color:black
style C fill:#bfb,color:black
style M fill:#bfb,color:black
style I fill:#bfb,color:black
style J fill:#bfb,color:black
style N fill:#bfb,color:black
style L fill:#bfb,color:black

A --> B
B --> C
C -->|Yes| M
C -->|No| J
M --> I
J --> L
I --> N

end

subgraph Release kubewarden charts 

s3{Is patch release?}
s4{All the components has the same released version?}
s5{Is policy-server release?}
s6[Update kubewarden-defaults chart version]
s8[Update kubewarden-controller chart version]
s14[Update kubewarden-crds chart version]
s7[Release kubewarden-defaults chart]
s9[Release kubewarden-controller chart]
s15[Release kubewarden-crds chart]
s10[Update kubewarden-defaults chart version]
s12[Update kubewarden-controller chart version]
s16[Update kubewarden-crds chart version]
s11[Release kubewarden-defaults chart]
s13[Release kubewarden-controller chart]
s17[Release kubewarden-crds chart]
s18[Finish]

style s3 fill:#bbf,color:black
style s4 fill:#bbf,color:black
style s5 fill:#bbf,color:black
style s6 fill:#bbf,color:black
style s7 fill:#bbf,color:black
style s8 fill:#bbf,color:black
style s9 fill:#bbf,color:black
style s10 fill:#bbf,color:black
style s11 fill:#bbf,color:black
style s12 fill:#bbf,color:black
style s13 fill:#bbf,color:black
style s14 fill:#bbf,color:black
style s15 fill:#bbf,color:black
style s16 fill:#bbf,color:black
style s17 fill:#bbf,color:black
style s18 fill:#bbf,color:black

s3 -->|No| s4
s4 -->|No| s18
s3 -->|Yes| s5
s5 -->|Yes| s6
s6 --> s7
s5 -->|No| s8
s8 --> s14
s14 --> s9
s9 --> s15
s4 -->|Yes| s10
s10 --> s12
s12 --> s16
s16 --> s11
s11 --> s13
s13 --> s17

end

subgraph Run E2E in the latest chart releases

e1[Run E2E tests in the latest chart releases]
e2{E2E test passed?}
e3[Notify Kubewarden team]
e4[Finish]

style e1 fill:#ffb,color:black
style e2 fill:#ffb,color:black
style e3 fill:#ffb,color:black
style e4 fill:#ffb,color:black

e1 --> e2
e2 -->|No| e3
e2 -->|Yes| e4
e3 --> e4

end

N -..-> s3
s17 -..-> e1
s7 -..-> e1
s15 -..-> e1

[jvanz]

Closing this issue. New issues to define what needs to be done will be created soon.