Sync conflict between multiclusterhub-operator and opp-policy ArgoCD Application

[lpanza]

There is a sync conflict between these two objects since the multiclusterhub-operator tries to add the "acm" ConsolePlugin while the "opp-policy" Application in ArgoCD tries to leave just the "odf-console" one

Here is the ArgoCD Application's Manifest for reference:

---
apiVersion: operator.openshift.io/v1
kind: Console
metadata:
  labels:
    argocd.argoproj.io/instance: opp-policy
  name: cluster
spec:
  plugins:
    - odf-console

Here are the operator logs:

1.6750967510576084e+09  INFO    Reconciling MultiClusterHub {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967510667846e+09  INFO    controller_multiclusterhub  Found image overrides from environment variables set by operand image prefix
1.6750967516626077e+09  INFO    Reconciling MultiClusterHub {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517205005e+09  INFO    Ensuring namespace: multicluster-engine {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517205572e+09  INFO    Ensuring operator group exists in ns: multicluster-engine   {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517205713e+09  INFO    Ensuring OLM multicluster-engine/multicluster-engine subscription   {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.675096751740915e+09   INFO    Lets preserve that namespace!   {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517757106e+09  INFO    MCH Current Version: '2.6.3', Desired Version: '2.6.3'  {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517757683e+09  INFO    Checking if current version is 2.6  {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517757752e+09  INFO    Checking if desired version is 2.6  {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517757795e+09  INFO    isACM26x=true, isUpgrade=false  {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517757983e+09  INFO    MCE current version '2.1.4', MCE desired version '2.1.4'    {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967517758012e+09  INFO    Checking MCE current version    {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.675096751775806e+09   INFO    MultiClusterEngine version '2.1.4' does not need to be updated  {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.675096751775812e+09   INFO    using trust bundle configmap open-cluster-management/trusted-ca-bundle  {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967526264307e+09  INFO    Dynamic plugins are supported. Adding ACM plugin to console {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967526264775e+09  INFO    Ready to add plugin {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}
1.6750967526500366e+09  INFO    Added acm consoleplugin to console  {"controller": "multiclusterhub", "controllerGroup": "operator.open-cluster-management.io", "controllerKind": "MultiClusterHub", "multiClusterHub": {"name":"multiclusterhub","namespace":"open-cluster-management"}, "namespace": "open-cluster-management", "name": "multiclusterhub", "reconcileID": "dfe729cc-f136-4936-9a62-824397619e25"}

[mbaldessari]

@day0hero @ipbabble Can you guys take a look?

[mbaldessari]

Thanks for the report @lpanza . For now this has been somewhat worked around via https://github.com/hybrid-cloud-patterns/multicluster-devsecops/commit/0e24e92b3c0eefae15615748c38674b72327a310 I suspect that once we upgrade to ArgoCD 2.5 we can just use server side apply for this plugin:

diff --git a/charts/hub/opp/templates/ocm-observability-bucket-claim.yaml b/charts/hub/opp/templates/ocm-observability-bucket-claim.yaml
index ccbfb14..a98cc29 100644
--- a/charts/hub/opp/templates/ocm-observability-bucket-claim.yaml
+++ b/charts/hub/opp/templates/ocm-observability-bucket-claim.yaml
@@ -11,6 +11,8 @@ apiVersion: operator.openshift.io/v1
 kind: Console
 metadata:
   name: cluster
+  annotations:
+    argocd.argoproj.io/sync-options: ServerSideApply=true
 spec:
   plugins:
   - odf-console

I am leaving this open just to track the above.