Provide a config file for the opcap check command, holding information such as pre-requisites for packages under tests and per-package test configuration

Not so much of a feature request, first maybe the start of a discussion.

Is your feature request related to a problem? Please describe.

1) It was already discussed how bundles don't possess all the information we require for testing them.

For example:

The operator requires multi-node, and therefore cannot be tested on SNO.
Some other resources must be created (often manually) so that this operator can be succesfully deployed.

For this second example, usually those pre-requisites are expressed in the bundle's description. This format (free text) doesn't allow us to automatically parse it and infer a list of resources to create before an operator is tested. AFAIK, our position so far is to implicitly exclude such operators for being tested. "If an operator requires other resources to run, then it cannot be tested by opcap".

2) In addition, it could be useful to provide per-package options to the opcap check command for certain audit.

For instance:

(once level 2 is implemented) I want to test the upgradability of an operator between version A and version B
I want to test additional Custom Resources with this operator. (this is already possible with the use of --extra-cr-directory)

Those options refer to a specific operator, whereas the opcap check command tests a list of operator.

Describe the solution you'd like.

I suggest having a yaml file describing the configuration options per operator. Something like:

operator1:
    requires: /path/to/manifest.yaml  <------- This contains all resources needed to be created before the operator is deployed
    is_sno_compatible: false
    audit_upgrade:
        from: v1.0.0
        to: v.1.1.0
    extra_CR:
        - /path/to/myCR1.yaml
        - /path/to/myCR2.yaml
operator2:
    requires: /path/to/another/manifest.yaml  <------- This contains all resources needed to be created before the operator is deployed
    is_sno_compatible: true
    audit_upgrade:
        from: v0.0.5
        to: v.0.0.6
    extra_CR:
        - /path/to/myCR1.yaml
        - /path/to/myCR2.yaml

(I haven't put a lot of thoughts in the structure itself, just a really basic example to show multiple operators with different requirements)

Describe alternatives you've considered.

Ideally, we would need to have as much information as we can present in the bundle itself. For example it would be much easier to be able to dynamically infer the pre-requisites from the bundle itself. So one alternative option would be to push changes upstream, but that's realistically not going to happen soon IMO.

As for the second part of the issue (per-package configuration), this could be resolved by having an equivalent of opcap check --packages myoperator i.e. a command to run a check on one single operator. In that case, it's trivial to add configuration options as flags in the CLI such as --audit-upgrade-from= and --audit-upgrade-to=

Additional context.

I think there are two questions that can be discussed and need to be answered: 1) Does it make sense to add those features at all ? I have the feeling that, as our audits become more complex, we might need those "per-package test options" more and more (we didn't start to think how to implement level 5, but I'm guessing that we'll need user inputs !) 2) What is the best way how to implement that ? The challenge is to keep it simple, which is the point on which I have mixed feelings ATM.

Now ~fight~ discuss !

opdev / opcap