search

SovereignCloudStack / issues

This repository is used for issues that are cross-repository or not bound to a specific repository.
https://github.com/orgs/SovereignCloudStack/projects/6
2 stars 1 forks source link

[Standard] CNCF conformance tests #636

Open tonifinger opened 1 month ago

tonifinger commented 1 month ago

This issue was created to provide a discussion ground for possible future standards. It is derived from #181 and one of the points not assigned any issue yet.

As a cloud service provider, I want my KaaS to comply with the Certified Kubernetes Software Conformance Here we discuss how the "Certified Kubernetes Software Conformance" can be utilized.

Definition of Done:

tonifinger commented 1 month ago

Reasearch:

Following the short guide, one can achieve CNCF certification by passing certain tests and submitting the test results to the CNCF Conformance Repo on GitHub (https://github.com/cncf/k8s-conformance, Requierments: Certified_Kubernetes_Terms.md)

The tests themselves can be carried out according to these instructions: https://github.com/cncf/k8s-conformance/blob/master/instructions.md .

Following these instructions, the tests are executed by sonobuoy, which furthermore executes the kubernetes e2e tests.

The SCS itself already uses these tests and tools within the k8s-cluster-api-provider The implementation of the tests is therefore already done when using the k8s-cluster-api-provider. In addition, I would suggest making it standard for the SCS KaaS to successfully pass the CNCF conformance tests.

List of certified products:

The CNCF provides a list of products that have earned a "certified kubernetes status" ( see: https://www.cncf.io/training/certification/software-conformance/#logos)

This list is divided into the following sections:

Perhaps the SCS could achieve to be listed in any of the products categories? Otherwise, we could write a decision dataset for users on what they need to do to get their SCS- deployed KaaS infrastructure certified by the CNCF.

SCS Implementation(draft):

Useful links:

mbuechse commented 1 month ago

@tonifinger See v2 here: https://docs.scs.community/standards/scs-compatible-kaas The question is: how do we transfer the results of the e2e pipeline into the compliance check tool?

tonifinger commented 2 weeks ago

@tonifinger See v2 here: https://docs.scs.community/standards/scs-compatible-kaas The question is: how do we transfer the results of the e2e pipeline into the compliance check tool?

To transfer the results of job/k8s-cluster-api-provider-e2e-conformance to the Python script scs-compliance-check.py we need to accomplish the following actions:

As a first step, the result of the conformance check is generated by the following “Zuul Job”: k8s-cluster-api-provider-e2e-conformance. ( The job executes the following playbook: playbooks/tasks/sonobouy.yaml )

Hint: The mode used in the Zuul job configuration is conformance (see: .zuul.yaml#L32) However in order to submit a result, according to "How to submit conformance results" we must use the mode certified-conformance

The generated data holding the results must be transferred to the "zuul job" running the "scs-compliance-check.py" script: k8s-cluster-api-provider-scs-compliance-1.27. ( The job executes the following playbook: playbooks/tasks/scs_compliance.yaml ) To accomplish this, we could use “zuul return-values”.

Finally, to check the results, the test script must be extended to analyze the test results generated by sonobuoy.

Hint: This goes hand in hand with the use of sonobouy as a test tool for the "kaas scs compliance tests": see kaas-sonobuoy-go-example-e2e-framework . We could use the same mechanism twice: First to analyze the "e2e conformance" test results Second to analyze the "kaas scs compliance tests" results

Question: Do we just want to test whether we would archive a certificate of compliance or do we also want to provide a mechanism to handle the process of archiving a certificate?