[Release] Check list for v0.6.0

[Xynnn007]

v0.6.0

Code freeze & release for tenant-componants (attestation-service/kbs)

Also, this release also includes KBS/AS. We might need the following steps:

• [x] 1. Cut an attestation-service v0.6.0 and make images for AS and RVPS, if changes happened in the project.

  * https://github.com/confidential-containers/attestation-service
  * Cut a release (AS/RVPS images will be automatically built triggered by release)

• [x] 2. Update kbs to use the latest commit from attestation-service, cut a release and make image PR: https://github.com/confidential-containers/kbs/pull/110

  * https://github.com/confidential-containers/kbs/blob/main/src/api_server/Cargo.toml
  * Change the revision for the following crates (both use `v0.6.0`)
   * `as-types`
   * `attestation-service`
  * Cut a release (kbs image will be automatically built triggered by release)

Note: If there are breaking changes in the communication protocol between KBS and KBC, consider replacing the image in CI with the AS/KBS image of the newly released.

Code freeze for guest-componants (attestation-agent/ocicrypt-rs/image-rs)

• [x] - 1. Update ocicrypt-rs to use the latest commit from attestation-agent @Xynnn007

  * https://github.com/confidential-containers/ocicrypt-rs/blob/main/Cargo.toml
    * Change the revision of both `attestation_agent` and `kbc`
    * Run `cargo update -p attestation_agent`
    * Run `cargo update -p kbc`

• [x] - 2. Update image-rs to use the latest commit from ocicrypt-rs and attestation-agent @Xynnn007

  * https://github.com/confidential-containers/image-rs/blob/main/Cargo.toml
    * Change the revision of `ocicrypt-rs` and `attestation_agent`
    * Run `cargo update -p ocicrypt-rs`
    * Run `cargo update -p attestation_agent`

• [x] - 3. Update Enclave CC to use the latest commit from image-rs PR: https://github.com/confidential-containers/enclave-cc/pull/162

      * https://github.com/confidential-containers/enclave-cc/blob/main/src/enclave-agent/Cargo.toml
        * Change the revision
        * Run `cargo update -p image-rs`
      Note that you can point to your own fork here, so you don't actually do changes in the other projects
      before making sure this step works as expected.

• [x] - 4. Update Kata Containers to use the latest commit from image-rs PR: https://github.com/kata-containers/tests/pull/5640 https://github.com/kata-containers/kata-containers/pull/6847

      * https://github.com/kata-containers/kata-containers/blob/CCv0/src/agent/Cargo.toml
        * Change the revision
        * Run `cargo update -p image-rs`
      Note that you can point to your own fork here, so you don't actually do changes in the other projects
      before making sure this step works as expected.

• [x] - 5. Update Kata Containers to use the latest attestation-agent PR: https://github.com/kata-containers/kata-containers/pull/7023

      * https://github.com/kata-containers/kata-containers/blob/CCv0/versions.yaml
        * Change the version

• [x] - 6. Update Kata Containers to use the latest td-shim PR: https://github.com/kata-containers/kata-containers/pull/7023

      * https://github.com/kata-containers/kata-containers/blob/CCv0/versions.yaml
        * Change the version

• [x] - 7. Check if there are new changes in the pre install payload script PR: https://github.com/confidential-containers/operator/pull/215

  * https://github.com/confidential-containers/operator/tree/main/install/pre-install-payload
    * The last commit there must match what's in the following files as preInstall / postUninstall image
      * Enclave CC: https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml
      * Kata Containers:
        Note that for Kata Containers, we're looking for the newTag, below the quay.io/confidential-containers/container-engine-for-cc-payload image
        * default: https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/default/kustomization.yaml

• [x] - 8. Ensure the Operator is using the latest CI builds and that the Operator tests are passsing PR: https://github.com/confidential-containers/operator/pull/215

  * Enclave CC:
    * SIM: https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/sim/kustomization.yaml
    * HW: https://github.com/confidential-containers/operator/blob/main/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml
    * Note that we need the quay.io/confidential-containers/runtime-payload-ci registry and enclave-cc-{SIM,HW}-latest tags
  * Kata Containers:
    * default: https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/default/kustomization.yaml
    * peer-pods: https://github.com/confidential-containers/operator/blob/main/config/samples/ccruntime/peer-pods/kustomization.yaml
    Note that we need the quay.io/confidential-containers/runtime-payload-ci registry and kata-containers-latest tag

• [x] - 9. Cut an attestation-agent v0.6.0, if changes happened in the project

• [x] - 10. Cut an ocicrypt-rs v0.6.0 release, using the latest release of: PR: https://github.com/confidential-containers/ocicrypt-rs/pull/67

  • attestation-agent (redo step 2, but now using v0.6.0)

• [x] - 11. Cut an image-rs v0.6.0 release, using the latest release of: PR: https://github.com/confidential-containers/image-rs/pull/157

  • ocicrypt-rs (redo step 1, but now using v0.6.0)
  • attestation-agent (redo step 2, but now using v0.6.0)

• [x] - 12. Cut a td-shim v0.6.0 release, if changes happened in the project

• [x] - 13. Update Enclave CC to use the released version of image-rs PR: https://github.com/confidential-containers/enclave-cc/pull/168

  • redo step 3, but now using v0.6.0

• [x] - 14. Update Kata Containers to the latest released version of: PR: https://github.com/kata-containers/kata-containers/pull/7037

  * image-rs (redo step 4, but now using the v0.6.0)
  * attestation-agent (redo step 5, but now using the v0.6.0)
  * td-shim (redo step 6, but now using the v0.6.0)

• [x] - 15. Update the operator to use the images generated from the latest commit of both Kata Containers and Enclave CC PR: https://github.com/confidential-containers/operator/pull/216

  * redo step 8, but now targetting the latest payload image generated for Kata Containers and Enclave CC

• [x] - 16. Make sure all the operator tests are passing

• [x] - 17. Cut an Enclave CC release

• [x] - 18. Add a new Kata Containers tag

• [x] - 19. Contact @stevenhorsman and/or @bpradipt to get the peer pods release ready

  • Update the versions of the peer pods dependencies:
  • attestation-agent - set to the the v0.6.0 tag
  • kata-containers - set to the CC-0.6.0 tag PR: https://github.com/confidential-containers/cloud-api-adaptor/pull/1059
  • Create the peer pods v0.6.0 release: https://github.com/confidential-containers/cloud-api-adaptor/releases/tag/v0.6.0
  • Wait for the release artifacts to be generated
  • Run the tests on the released artifacts
  • Update any peer pods release notes
  • Update go mod to point to released version for peerpod-ctl and csi-wrapper e.g. confidential-containers/cloud-api-adaptor/pull/825

Release

• [x] - 20. Update the operator to use the release tags coming from Enclave CC and Kata Containers PR: https://github.com/confidential-containers/operator/pull/217

  * redo step 8, but now targetting thje latest release of the payload image generated for Kata Containers eand Enclave CC

• [x] - 21. Update the Operator version PR: https://github.com/confidential-containers/operator/pull/217

  * https://github.com/confidential-containers/operator/blob/main/config/release/kustomization.yaml#L7

• [x] - 22. Cut an operator release

• [x] - 23. Make sure to update the release notes PR: https://github.com/confidential-containers/documentation/pull/124

  * https://github.com/confidential-containers/documentation/tree/main/releases/v0.6.0.md

• [ ] - 24. Poke Jens Freimann (jfreiman@redhat.com) to update the release to the OperatorHub

[Xynnn007]

Change the order slightly, because CI may use the new image that has just been released

[zvonkok]

I would like to add https://github.com/kata-containers/kata-containers/pull/6993 for the CC GPU use-case.

[zvonkok]

On which version of Kata will 0.6.0 be based? I also want to make sure we have https://github.com/kata-containers/kata-containers/pull/6699 in the 0.6.0

[fitzthum]

@stevenhorsman This checklist doesn't contain anything regarding peer pods. What is the process for the release there and are there things we should be adding to the checklist?

[stevenhorsman]

@stevenhorsman This checklist doesn't contain anything regarding peer pods. What is the process for the release there and are there things we should be adding to the checklist?

Yeah, I'll add an extra section here for it. Thanks for the reminder

[fitzthum]

@zvonkok Release v0.6.0 will use a bundle built from the CCv0 branch of Kata (see step 15). So changes for this release may need to get backported to CCv0.

This checklist is really for tasks that facilitate the release itself. If there are more things to get in before the freeze starts, then your best bet is to get them merged before EOD today. Otherwise it's a question of whether to delay the freeze, which would need to be a wider discussion.

Hopefully the next release will be based on main, so if things don't make it into this one it should be simpler next time around.

[fitzthum]

Ok, I think we will need to fix this for the release. https://github.com/confidential-containers/kbs/issues/102 The PR https://github.com/confidential-containers/attestation-service/pull/38 should fix this (among other things) so I will try to find a reviewer for that asap.

[sprt]

Can't tag myself but I'll update the Kata Containers dependencies (4-6, 14).

[bpradipt]

@Xynnn007 it'll be good if KBS image is pushed to quay.io/confidential-containers registry like the other images.

[Xynnn007]

@Xynnn007 it'll be good if KBS image is pushed to quay.io/confidential-containers registry like the other images.

Yes @bpradipt , the release CI will do this, and I hope to cut release for KBS after release guest components.

The reason is that today I found that kbs refers to AA' code https://github.com/confidential-containers/kbs/blob/main/tools/client/Cargo.toml#L16, so my idea is to cut release for the guest components and then change the rev inside that Cargo.toml.

BTW, as currently only tdx has been using kbs in CI afaik, Arron has helped update the tdx ci image already. Thus we do not need to worry about the CI in kata

[fitzthum]

Regarding number 7, the s390 and default operator yamls both point to the latest tag of container-engine-for-cc-payload. Does that seem correct to you @stevenhorsman or should we be fixing the yaml to a particular hash for the release? We haven't updated the preInstall container in two months anyway.

For enclave-cc, we are not pointing to the latest preInstall image (see here). This is also not the most recent hash. Is this intentional @mythi or should this also be updated to point to latest?

[mythi]

For enclave-cc, we are not pointing to the latest preInstall image (see here). This is also not the most recent hash. Is this intentional @mythi or should this also be updated to point to latest?

It's not intentional, thanks for noticing! I had missed that other CcRuntimes had moved to kustomization for these too. However, it looks the containerd version has not changed since 0.5.0 which triggers the question have these images been updated?

[stevenhorsman]

Regarding number 7, the s390 and default operator yamls both point to the latest tag of container-engine-for-cc-payload. Does that seem correct to you @stevenhorsman or should we be fixing the yaml to a particular hash for the release? We haven't updated the preInstall container in two months anyway.

Yes - I agree that we should switch it back to a proper hash for the release. I suspect that it wasn't worth my switching it to latest when I did the same for the runtime payload to be honest as we don't update that payload often enough, so I'll try and remember to not do that next time.

The other thing to note is that when I made the runtime-payload multi-arch I switched the s390x to base on default, so I don't think we need the specific s390x update any more - I updated part 8 in the checklist so remove it, but forget part 7, so apologies for any confusion and I'll update the template once the release has gone out.

I hope that helps and answers the question?

[fitzthum]

Ok, just waiting for the payload build to finish and then I will do 7 and 8.

[fitzthum]

Nvm, we have https://github.com/confidential-containers/operator/pull/215

[stevenhorsman]

Nvm, we have confidential-containers/operator#215

Sorry Tobin, I should have mentioned on here I was looking at it.

[fitzthum]

No worries. Thanks for the PR.

[stevenhorsman]

@Xynnn007 - do you know if anyone is working on creating the releases of the components (steps 9-12), or is help needed there?

[Xynnn007]

@stevenhorsman AFAIK, no. I can do 10 & 11, while I do not have permission in AA (9) and td-shim (12). Could you help with this?

[stevenhorsman]

AFAIK, no. I can do 10 & 11, while I do not have permission in AA (9) and td-shim (12). Could you help with this?

Sure - I can help with which ever numbers you like and @fitzthum has a github team for release-champions I believe that would give you access to create the tags.

[fitzthum]

Yes, I will add both of you to the release champions group. Hopefully we can get the tagging and bumping done today and then @sprt can do 14 this afternoon. I can also do some of those steps if you want to go to sleep @Xynnn007

[stevenhorsman]

Just as an update on 12 - td-shim has had two commits added since we tested it in the release candidate. I'm hoping that they aren't required, but I've messaged Jiewen to confirm before we cut the release - but if anyone else knows before we get the update that would help us move onto the next steps quicker.

[stevenhorsman]

Just as an update on 12 - td-shim has had two commits added since we tested it in the release candidate. I'm hoping that they aren't required, but I've messaged Jiewen to confirm before we cut the release - but if anyone else knows before we get the update that would help us move onto the next steps quicker.

FYI - we cut https://github.com/confidential-containers/td-shim/releases/tag/v0.6.0 based on 3252047 , but if the extra commits are required we will create a v0.6.1 release tomorrow.

[stevenhorsman]

I've done up to step 13, which has the PR https://github.com/confidential-containers/enclave-cc/pull/168 in review. Over to @sprt for step 14 now :)

[sprt]

@Xynnn007 PR for step 14: https://github.com/kata-containers/kata-containers/pull/7037

[stevenhorsman]

Step 16 tests all passed, so I think we are ready to release enclave-cc & tag kata-containers. Are there already people signed up to look at those tasks?

[Xynnn007]

Step 16 tests all passed, so I think we are ready to release enclave-cc & tag kata-containers. Are there already people signed up to look at those tasks?

Let me cut enclave-cc

[fitzthum]

Who has permissions to make tag for Kata?

[fitzthum]

Should we just update https://github.com/confidential-containers/operator/pull/216 to use the tags once we have them?

[stevenhorsman]

I think the Kata tag has worked correctly: https://github.com/kata-containers/kata-containers/releases/tag/CC-0.6.0

[stevenhorsman]

BTW - I'm working on 19 now

[fitzthum]

We may have a minor issue with enclave-cc release workflow that we need https://github.com/confidential-containers/enclave-cc/pull/170 to fix

[stevenhorsman]

FYI - I did the peer pod release and have tested it on IBM Cloud and it all worked. I'm going to wait for someone to do AKS testing before I formerly declare that it's successful, but 19 is at least 75% ticked :)

[fitzthum]

@jensfr can you take a look at step 24?

[fitzthum]

I have created an issue with some reflections https://github.com/confidential-containers/community/issues/95

[stevenhorsman]

Hey folks, I just want to clarify/tie up a loose end on the operator. There was a bug in the operator hub install that got fixed yesterday (with https://github.com/confidential-containers/operator/pull/214), after the v0.6.0 release was cut. Do we want to cut a v0.6.1 which is just v0.6.0 + that fix before we get Jens to update operator hub?

[fitzthum]

I think that is a good idea. No point in releasing v0.6.0 on the operator hub.

[stevenhorsman]

I think that is a good idea. No point in releasing v0.6.0 on the operator hub.

Cool - do you want me to cut the new release, or someone else signed up? If we do it today then it will be ready for Jens when he is back from PTO!

[fitzthum]

Nobody is signed up. Feel free to make the release.

[stevenhorsman]

Nobody is signed up. Feel free to make the release.

Done: @jensfr - https://github.com/confidential-containers/operator/releases/tag/v0.6.1 should be good for operator hub when you are ready

[Xynnn007]

Can we close this issue now?

[stevenhorsman]

Can we close this issue now?

@jensfr - are you able to update operator hub is it looks like it is still on version 0.5.0 from https://operatorhub.io/operator/cc-operator?