Design: oadp operator CRD validation ( improved user experience w/ multiple velero installs)

[weshayutin]

Customers often have more than one install of OADP on their cluster. They will often have OADP installed for backup and restore and ACM installed for cluster managment. Often these two products are installed at different times and both installs work, however they are not at the expected OADP version in one of the installs.

We need to find an appropriate way to warn the user that there is an operator / oadp crd mismatch on the cluster.

[mateusoliveira43]

@weshayutin please add a description to the issue or close it

[weshayutin]

ah agree... k updating

[kaovilai]

Thanks for the update!

[mateusoliveira43]

There is no way to be sure if OADP CRDs mismatch only with cluster information.

OADP could warn users (in logs, events and/or DPA Status) that it might happen based on the following information:

• CRD metadata annotation operatorframework.io/installed-alongside-<random>: <oadp-namespace>/oadp-operator.v<oadp-version> matches OADP CSV version

CRD also have generation metadata, but after upgrades, checking if is equals 1, can give false positives as well.

Only way to be sure, is to check if CRD YAML in cluster matches that OADP release CRD YAML, which can not be done always, because some cluster environment are disconnected. But user can dump information and compare, to confirm.

[kaovilai]

Perhaps we can copy expected CRD into Dockerfile as part of build, and operator can compare container image CRD to the in cluster CRD which we can call "oc get" on.

[kaovilai]

It should work IMO as long as operator has permission to call get on CRD.

[shubham-pampattiwar]

I think we can keep this simple, We should check two things:

• CRD version number via CRD annotations: operatorframework.io/installed-alongside-be8e31d62616a31f: openshift-adp/oadp-operator.v99.0.0 e.g:

  spampatt@spampatt-mac oadp-operator [master]$ oc get crd backups.velero.io -oyaml | grep operatorframework.io/installed-alongside
  operatorframework.io/installed-alongside-be8e31d62616a31f: openshift-adp/oadp-operator.v99.0.0

• Operator version number from oadp-operator csv spec.version

  spampatt@spampatt-mac oadp-operator [master]$ oc get csv -n openshift-adp oadp-operator.v99.0.0 -oyaml | grep 'version: 99'
  version: 99.0.0

  Now if these 2 things don't match then add a warning on DPA CR. We could add this as part of the preflight validation checks in DPA reconciler. This could the first task in the DPA reconcile loop executes once DPA is created by the user and if there is a CRD : Operator version mismatch, just add a warning log/DPA warning or maybe both.

[kaovilai]

Ok that would avoid spec diff check.. but may not be absolutely correct if CRD was manually installed the right version.. as it sometime is via ACM backup "chart".

[shubham-pampattiwar]

I think there must some label/annotation/identifier that we can use from CRDs in ACM case. If not, then at the very least we could ask ACM folks to add it to their CRD yamls.

[kaovilai]

works

[sseago]

Sounds like there are 2 things we may want to do here: 1) Compare CRD version in-cluster to expected CRD version -- this would be an ongoing operator reconcile, so that if someone later installed the wrong CRDs as part of another velero/OADP install, current OADP would warn or error out. 2) Some form of notification at install time that warns a user that other OADP versions are already in the cluster.

[weshayutin]

@shubham-pampattiwar I think this is a great comment and improvment. We'll disucss more, but THANK YOU!! for the creative solution RE: https://github.com/openshift/oadp-operator/issues/1134#issuecomment-2392637925