search

Azure / container-upstream

This project captures work in progress, and completed work for the Azure Core Container Upstream team
MIT License
54 stars 28 forks source link

Automate generating conformance results for CAPZ certification #121

Closed CecileRobertMichon closed 1 year ago

CecileRobertMichon commented 2 years ago

We should upload conformance results for CAPZ self-managed k8s to https://github.com/cncf/k8s-conformance for each k8s version so it can be certified as a conformant k8s distribution.

To avoid having to do it manually each time, we should consider building a pipeline to automate running conformance and extracting logs/results.

See https://dev.azure.com/AzureContainerUpstream/Kubernetes/_git/kubernetes-tests?path=/pipelines/kubernetes.sonobuoy.yml&version=GBchuwon/add-more-tests for prior art

dtzar commented 1 year ago

Having run this manually, one hurdle is going to be to handle the fact that the test itself sonobuoy run --mode=certified-conformance runs in the background for an hour or more.

dtzar commented 1 year ago

We decided to not do conformance testing for CAPZ, but only for AKS. If we get any customer requests for this, we can reconsider.

ritazh commented 1 year ago

Originally this issue was added to ensure Microsoft has at least one conformance result uploaded to https://github.com/cncf/k8s-conformance for each k8s version so it can be certified as a conformant k8s distribution.

@yizhang4321 Can you please confirm if AKS now has automation to ensure for each new k8s version that is released upstream, AKS generates and submits conformance results to https://github.com/cncf/k8s-conformance?

yizhang4321 commented 1 year ago

When AKS supports a new Kubernetes minor version, say 1.26, we have process in place to make sure the conformance result is uploaded. For example, for 1.26, we had this:

https://github.com/cncf/k8s-conformance/pull/2443

We currently have process in place to make sure this happens, but it's not fully automated yet.

lzhecheng commented 1 year ago

@yizhang4321 I have created a pipeline for aks certificated. Does it meet your expectation? It is mentioned in the email thread: "Quality assurance and checklist for releasing new k8s versions"