stolostron / multicloud-operators-deployable

Multicloud application management with a deployable CRD.
Apache License 2.0
6 stars 12 forks source link

Deployable status fails to propagate back to ACM hub with StorageError: invalid object, Code: 4, Key:XXXXXXXX #53

Open arivatibm opened 4 years ago

arivatibm commented 4 years ago

I'm getting this type of error : StorageError: invalid object, Code: 4, Key:xxxxxx

This happens when deploying a Deployable directly to ACM hub Note that CR deployment reaches the managed cluster, the CR is deployed correctly but its status does not propagate back to ACM hub due to error below

The same issue happens when I deploy to 3 different managed openshift clusters I tried to delete the Key from ETCD that the error is complaining about - but the error keeps persisting

Could you assist?

apiVersion: apps.open-cluster-management.io/v1
kind: Deployable
metadata:
  annotations:
    open-cluster-management.io/user-group: c3lzdGVtOmF1dGhlbnRpY2F0ZWQ6b2F1dGgsc3lzdGVtOmF1dGhlbnRpY2F0ZWQ=
    open-cluster-management.io/user-identity: QVJJVkBpbC5pYm0uY29t
  selfLink: >-
    /apis/apps.open-cluster-management.io/v1/namespaces/ariv-source/deployables/example-ippbx-1-dpl
  resourceVersion: '5994574'
  name: example-ippbx-1-dpl
  uid: 1172728f-3a25-4a87-8cc6-65c44bfb1493
  creationTimestamp: '2020-11-08T14:44:25Z'
  generation: 1
  namespace: ariv-source
  labels:
    appname: example
    crname: example-ippbx-1
spec:
  placement:
    placementRef:
      kind: PlacementRule
      name: example-ippbx-1-placementrule
  template:
    apiVersion: ippbx.example.com/v1alpha1
    kind: Ippbx
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: >
          {"apiVersion":"ippbx.example.com/v1alpha1","kind":"Ippbx","metadata":{"annotations":{},"labels":{"kubedriver.alm/deploymentLocation":"edge1"},"name":"example-ippbx-1","namespace":"ariv-source"},"spec":{}}
      selfLink: >-
        /apis/ippbx.example.com/v1alpha1/namespaces/ariv-source/ippbxes/example-ippbx-1
      resourceVersion: '5993196'
      name: example-ippbx-1
      uid: 091d621a-ac67-4bc0-8a84-a01cff408fd1
      creationTimestamp: '2020-11-08T14:43:29Z'
      generation: 1
      namespace: ariv
      labels:
        kubedriver.alm/deploymentLocation: edge1
    spec: {}
status:
  lastUpdateTime: '2020-11-08T14:45:53Z'
  phase: Propagated
  targetClusters:
    edge1:
      lastUpdateTime: '2020-11-08T14:45:53Z'
      phase: Failed
      reason: >-
        Operation cannot be fulfilled on ippbxes.ippbx.example.com
        "example-ippbx-1": StorageError: invalid object, Code: 4, Key:
        /kubernetes.io/ippbx.example.com/ippbxes/ariv/example-ippbx-1,
        ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in
        precondition: 091d621a-ac67-4bc0-8a84-a01cff408fd1, UID in object meta:
        87490556-c086-42a4-a5b9-b4838b2ec734
      resourceStatus:
        conditions:
          - ansibleResult:
              changed: 2
              completion: '2020-11-08T14:45:14.76937'
              failures: 1
              ok: 3
              skipped: 1
            lastTransitionTime: '2020-11-08T14:45:15Z'
            message: unknown playbook failure
            reason: Failed
            status: 'False'
            type: Failure
          - lastTransitionTime: '2020-11-08T14:45:15Z'
            message: Running reconciliation
            reason: Running
            status: 'True'
            type: Running 
arivatibm commented 4 years ago

Getting this same error on the appmgr pod in edge1 cluster: E1109 11:21:48.504239 1 synchronizer.go:276] Failed to update resource with error:Operation cannot be fulfilled on ippbxes.ippbx.example.com "example-ippbx-1": StorageError: invalid object, Code: 4, Key: /kubernetes.io/ippbx.example.com/ippbxes/ariv/example-ippbx-1, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: a6ba3ded-34ad-4683-9dff-27f75f62944e, UID in object meta: 87490556-c086-42a4-a5b9-b4838b2ec734