Opcap's goal has been changed to be the tool for testing if OpenShift changes won't break a big number of operators at once. Specially the infrastructure operators. And in multiple different OCP flavors. With that the capability levels are no longer on the road map for the near future. In simple lines we could discard a good amount of code and keep it as simple as we possibly can. Here goes a suggestion on the new workflow for the tool on big lines:
Choose a catalog source
Install each operator followed by the operand (we may consider no longer run operators alone)
grab the basic results with (Suceed|Failed|Timeout) for operators and (Created|Failed) for operands and report
collect the debug data such as events and logs to identify the origin of issues and make sure it's not OpenShift's fault
Report when an OpenShift change breaks something.
One thing that could be interesting is adding the reason of failures and timeouts to the report. That requires work on detailed debug data collection.
Flags I think we no longer need:
CatalogSourceNamespace - we can hard code it with openshift-marketplace
auditPlan - we can hard code it to run always operator/operand
all-installmodes - Not sure if we need to have this one. If we test with a supported install mode it should be good to go
Other changes I would propose:
Eliminating auditPlan and just chain the tests on a simple slice.
Detaching the workqueue out of it's struct and use it as a simple channel variable
Building the options out of a simple struct on the check command with no functions attached to that like the WithAuditPlan. Instead we just pass a filled in struct and we can write code on the capability package to validate it if needed.
Rename the capability package to just opcap package and merge it with the operator, packages, bundle
Rename AuditorOptions to just options
Rename auditOptions and capAudit to just audit and merge them in a single struct
Add a field in the audit struct to hold the now named options from the auditorOptions, this way we can hold all general options and always passe it away to the next function in line.
With just a simple slice to hold tests we no longer need a stack and the order of running and cleaning should be straightforward and hard coded.
Evaluate the possibility of merging some of the internal package while eliminating lines of code with and repeated client initiation.
Keep as less files as possible while reducing the number of lines too.
Since we don't care about reporting from a database or even from the screen eliminate those options and use a simple json marshaller to write the json files. I think there is no need for a report package. It's a file generator function.
Opcap's goal has been changed to be the tool for testing if OpenShift changes won't break a big number of operators at once. Specially the infrastructure operators. And in multiple different OCP flavors. With that the capability levels are no longer on the road map for the near future. In simple lines we could discard a good amount of code and keep it as simple as we possibly can. Here goes a suggestion on the new workflow for the tool on big lines:
One thing that could be interesting is adding the reason of failures and timeouts to the report. That requires work on detailed debug data collection.
Flags I think we no longer need:
Other changes I would propose: