enhancement to add test metadata into the bundle

operator-framework / enhancements

Apache License 2.0

9 stars 40 forks source link

enhancement to add test metadata into the bundle #18

Closed jmccormick2001 closed 4 years ago

jmccormick2001 commented 4 years ago

This PR is an enhancement to allow test metadata to be added into a bundle, this will enable testing against a bundle. For operator-sdk scorecard development underway, we would make use of this feature to co-locate tests with the operator metadata. End users would use this feature to add test metadata, update test metadata within their operator bundles. Test frameworks would need an ability to extract test metadata from bundles when executing tests.

estroz commented 4 years ago

From offline discussion:

This PR should act as a guide for both future additive bundle specs and a set of "rules" for creating such a spec. The latter could be an addendum to the enhancement, or a separate doc in its own right.
Because the bundle format is extensible, there should be an abstraction over bundle.Dockerfile and annotations.yaml that allows label modification without giving users too much freedom to modify bundles. Not sure what that looks like yet, but it will likely involve some interface work in operator-registry.

jmccormick2001 commented 4 years ago

I updated the enhancement to add the proposed annotations plus mention that operator-registry would need to have an API to add this sort of metadata into a bundle.

jmccormick2001 commented 4 years ago

I updated this to show how a test framework other than scorecard would show up in the label nomenclature just as an example.

jmccormick2001 commented 4 years ago

I removed the annotation that specified kuttl, I didn't see what its purpose would be in the context of scorecard. Scorecard will just process tests as it finds them within the scorecard metadata essentially, regardless if they are kuttl or not. The only value I could think of in having a kuttl annotation is if some other tooling outside of scorecard cared about what types of scorecard tests were being run, I'm not aware of that being a requirement at this point.

jmccormick2001 commented 4 years ago

@ecordell wdyt? did I miss any of your concerns/questions or items I need to address?

ecordell commented 4 years ago

/lgtm

joelanford commented 4 years ago

/lgtm

shawn-hurley commented 4 years ago

/lgtm

I am a little worried that the user stories may require more work and design. We should consider updating this enhancement as we implement them.