search

red-hat-storage / ocs-ci

https://ocs-ci.readthedocs.io/en/latest/
MIT License
108 stars 166 forks source link

test_mds_cache_alert_with_all_mds_node_scaledown leftovers on tierdown #10590

Closed DanielOsypenko closed 4 days ago

DanielOsypenko commented 1 month ago

We need to adjust test to remove leftovers or open a bug

https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/42472/consoleText

[2024-09-26T16:07:53.711Z] 12:07:53 - MainThread - ocs_ci.framework.pytest_customization.reports - INFO  - duration reported by tests/functional/monitoring/prometheus/alerts/test_alert_mds_cache_high_usage.py::TestMdsMemoryAlerts::test_mds_cache_alert_with_all_mds_node_scaledown immediately after test execution: 1661.01
[2024-09-26T16:07:53.711Z] 
[2024-09-26T16:07:53.711Z] tests/functional/monitoring/prometheus/alerts/test_alert_mds_cache_high_usage.py::TestMdsMemoryAlerts::test_mds_cache_alert_with_all_mds_node_scaledown ERROR
[2024-09-26T16:07:53.711Z] _ ERROR at teardown of TestMdsMemoryAlerts.test_mds_cache_alert_with_all_mds_node_scaledown _
[2024-09-26T16:07:53.711Z] 
[2024-09-26T16:07:53.711Z] exclude_labels = ['app=must-gather', 'app=s3cli', 'must-gather-helper-pod=']
[2024-09-26T16:07:53.711Z] 
[2024-09-26T16:07:53.711Z]     def get_status_after_execution(exclude_labels=None):
[2024-09-26T16:07:53.711Z]         """
[2024-09-26T16:07:53.711Z]         Set the environment status and assign it into ENV_STATUS_PRE dictionary.
[2024-09-26T16:07:53.711Z]         In addition compare the dict before the execution and after using DeepDiff
[2024-09-26T16:07:53.711Z]     
[2024-09-26T16:07:53.711Z]         Args:
[2024-09-26T16:07:53.711Z]             exclude_labels (list): App labels to ignore leftovers
[2024-09-26T16:07:53.711Z]     
[2024-09-26T16:07:53.711Z]         Raises:
[2024-09-26T16:07:53.711Z]              ResourceLeftoversException: In case there are leftovers in the
[2024-09-26T16:07:53.711Z]                 environment after the execution
[2024-09-26T16:07:53.711Z]         """
[2024-09-26T16:07:53.711Z]         get_environment_status(config.RUN["ENV_STATUS_POST"], exclude_labels=exclude_labels)
[2024-09-26T16:07:53.711Z]     
[2024-09-26T16:07:53.711Z]         pod_diff = compare_dicts(
[2024-09-26T16:07:53.711Z]             config.RUN["ENV_STATUS_PRE"]["pod"], config.RUN["ENV_STATUS_POST"]["pod"]
[2024-09-26T16:07:53.711Z]         )
[2024-09-26T16:07:53.711Z]         sc_diff = compare_dicts(
[2024-09-26T16:07:53.711Z]             config.RUN["ENV_STATUS_PRE"]["sc"], config.RUN["ENV_STATUS_POST"]["sc"]
[2024-09-26T16:07:53.711Z]         )
[2024-09-26T16:07:53.711Z]         pv_diff = compare_dicts(
[2024-09-26T16:07:53.711Z]             config.RUN["ENV_STATUS_PRE"]["pv"], config.RUN["ENV_STATUS_POST"]["pv"]
[2024-09-26T16:07:53.711Z]         )
[2024-09-26T16:07:53.711Z]         pvc_diff = compare_dicts(
[2024-09-26T16:07:53.711Z]             config.RUN["ENV_STATUS_PRE"]["pvc"], config.RUN["ENV_STATUS_POST"]["pvc"]
[2024-09-26T16:07:53.711Z]         )
[2024-09-26T16:07:53.711Z]         namespace_diff = compare_dicts(
[2024-09-26T16:07:53.711Z]             config.RUN["ENV_STATUS_PRE"]["namespace"],
[2024-09-26T16:07:53.711Z]             config.RUN["ENV_STATUS_POST"]["namespace"],
[2024-09-26T16:07:53.711Z]         )
[2024-09-26T16:07:53.711Z]         volumesnapshot_diff = compare_dicts(
[2024-09-26T16:07:53.711Z]             config.RUN["ENV_STATUS_PRE"]["vs"], config.RUN["ENV_STATUS_POST"]["vs"]
[2024-09-26T16:07:53.711Z]         )
[2024-09-26T16:07:53.711Z]         if config.RUN["cephcluster"]:
[2024-09-26T16:07:53.711Z]             cephfs_diff = compare_dicts(
[2024-09-26T16:07:53.711Z]                 config.RUN["ENV_STATUS_PRE"]["cephfs"],
[2024-09-26T16:07:53.711Z]                 config.RUN["ENV_STATUS_POST"]["cephfs"],
[2024-09-26T16:07:53.711Z]             )
[2024-09-26T16:07:53.711Z]             cephbp_diff = compare_dicts(
[2024-09-26T16:07:53.711Z]                 config.RUN["ENV_STATUS_PRE"]["cephbp"],
[2024-09-26T16:07:53.711Z]                 config.RUN["ENV_STATUS_POST"]["cephbp"],
[2024-09-26T16:07:53.711Z]             )
[2024-09-26T16:07:53.711Z]             diffs_dict = {
[2024-09-26T16:07:53.711Z]                 "pods": pod_diff,
[2024-09-26T16:07:53.711Z]                 "storageClasses": sc_diff,
[2024-09-26T16:07:53.711Z]                 "cephfs": cephfs_diff,
[2024-09-26T16:07:53.711Z]                 "cephbp": cephbp_diff,
[2024-09-26T16:07:53.711Z]                 "pvs": pv_diff,
[2024-09-26T16:07:53.711Z]                 "pvcs": pvc_diff,
[2024-09-26T16:07:53.711Z]                 "namespaces": namespace_diff,
[2024-09-26T16:07:53.711Z]                 "vs": volumesnapshot_diff,
[2024-09-26T16:07:53.712Z]             }
[2024-09-26T16:07:53.712Z]         elif config.RUN["lvm"]:
[2024-09-26T16:07:53.712Z]             lv_diff = compare_dicts(
[2024-09-26T16:07:53.712Z]                 config.RUN["ENV_STATUS_PRE"]["lv"],
[2024-09-26T16:07:53.712Z]                 config.RUN["ENV_STATUS_POST"]["lv"],
[2024-09-26T16:07:53.712Z]             )
[2024-09-26T16:07:53.712Z]             diffs_dict = {
[2024-09-26T16:07:53.712Z]                 "pods": pod_diff,
[2024-09-26T16:07:53.712Z]                 "storageClasses": sc_diff,
[2024-09-26T16:07:53.712Z]                 "pvs": pv_diff,
[2024-09-26T16:07:53.712Z]                 "pvcs": pvc_diff,
[2024-09-26T16:07:53.712Z]                 "namespaces": namespace_diff,
[2024-09-26T16:07:53.712Z]                 "vs": volumesnapshot_diff,
[2024-09-26T16:07:53.712Z]                 "lv": lv_diff,
[2024-09-26T16:07:53.712Z]             }
[2024-09-26T16:07:53.712Z]     
[2024-09-26T16:07:53.712Z]         leftover_detected = False
[2024-09-26T16:07:53.712Z]     
[2024-09-26T16:07:53.712Z]         leftovers = {"Leftovers added": [], "Leftovers removed": []}
[2024-09-26T16:07:53.712Z]         for kind, kind_diff in diffs_dict.items():
[2024-09-26T16:07:53.712Z]             if not kind_diff:
[2024-09-26T16:07:53.712Z]                 continue
[2024-09-26T16:07:53.712Z]             if kind_diff[0]:
[2024-09-26T16:07:53.712Z]                 leftovers["Leftovers added"].append({f"***{kind}***": kind_diff[0]})
[2024-09-26T16:07:53.712Z]                 leftover_detected = True
[2024-09-26T16:07:53.712Z]             if kind_diff[1]:
[2024-09-26T16:07:53.712Z]                 leftovers["Leftovers removed"].append({f"***{kind}***": kind_diff[1]})
[2024-09-26T16:07:53.712Z]                 leftover_detected = True
[2024-09-26T16:07:53.712Z]         if leftover_detected:
[2024-09-26T16:07:53.712Z] >           raise exceptions.ResourceLeftoversException(
[2024-09-26T16:07:53.712Z]                 f"\nThere are leftovers in the environment after test case:"
[2024-09-26T16:07:53.712Z]                 f"\nResources added:\n{yaml.dump(leftovers['Leftovers added'])}"
[2024-09-26T16:07:53.712Z]                 f"\nResources "
[2024-09-26T16:07:53.712Z]                 f"removed:\n {yaml.dump(leftovers['Leftovers removed'])}"
[2024-09-26T16:07:53.712Z]             )
[2024-09-26T16:07:53.712Z] E           ocs_ci.ocs.exceptions.ResourceLeftoversException: 
[2024-09-26T16:07:53.712Z] E           There are leftovers in the environment after test case:
[2024-09-26T16:07:53.712Z] E           Resources added:
[2024-09-26T16:07:53.712Z] E           []
[2024-09-26T16:07:53.712Z] E           
[2024-09-26T16:07:53.712Z] E           Resources removed:
[2024-09-26T16:07:53.712Z] E            - '***pods***':
[2024-09-26T16:07:53.712Z] E             - apiVersion: v1
[2024-09-26T16:07:53.712Z] E               kind: Pod
[2024-09-26T16:07:53.712Z] E               metadata:
[2024-09-26T16:07:53.712Z] E                 annotations:
[2024-09-26T16:07:53.712Z] E                   k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.129.0.45/23"],"mac_address":"0a:58:0a:81:00:2d","gateway_ips":["10.129.0.1"],"routes":[{"dest":"10.128.0.0/14","nextHop":"10.129.0.1"},{"dest":"172.30.0.0/16","nextHop":"10.129.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.129.0.1"}],"ip_address":"10.129.0.45/23","gateway_ip":"10.129.0.1"}}'
[2024-09-26T16:07:53.712Z] E                   k8s.v1.cni.cncf.io/network-status: "[{\n    \"name\": \"ovn-kubernetes\",\n\
[2024-09-26T16:07:53.712Z] E                     \    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.129.0.45\"\n\
[2024-09-26T16:07:53.712Z] E                     \    ],\n    \"mac\": \"0a:58:0a:81:00:2d\",\n    \"default\": true,\n \
[2024-09-26T16:07:53.712Z] E                     \   \"dns\": {}\n}]"
[2024-09-26T16:07:53.712Z] E                   openshift.io/scc: rook-ceph
[2024-09-26T16:07:53.712Z] E                 creationTimestamp: '2024-09-26T07:41:41Z'
[2024-09-26T16:07:53.712Z] E                 generateName: rook-ceph-osd-prepare-9afb3985e0230a85d5dd03e691b9c20d-
[2024-09-26T16:07:53.712Z] E                 labels:
[2024-09-26T16:07:53.712Z] E                   app: rook-ceph-osd-prepare
[2024-09-26T16:07:53.712Z] E                   batch.kubernetes.io/controller-uid: 853f612d-fc0e-48d8-b2bb-735be68a74a0
[2024-09-26T16:07:53.712Z] E                   batch.kubernetes.io/job-name: rook-ceph-osd-prepare-9afb3985e0230a85d5dd03e691b9c20d
[2024-09-26T16:07:53.712Z] E                   ceph.rook.io/DeviceSet: ocs-deviceset-gp3-csi-0
[2024-09-26T16:07:53.712Z] E                   ceph.rook.io/pvc: ocs-deviceset-gp3-csi-0-data-0l529m
[2024-09-26T16:07:53.712Z] E                   controller-uid: 853f612d-fc0e-48d8-b2bb-735be68a74a0
[2024-09-26T16:07:53.712Z] E                   job-name: rook-ceph-osd-prepare-9afb3985e0230a85d5dd03e691b9c20d
[2024-09-26T16:07:53.712Z] E                   rook_cluster: odf-storage
[2024-09-26T16:07:53.712Z] E                 name: rook-ceph-osd-prepare-9afb3985e0230a85d5dd03e691b9c20d-bjkkn
[2024-09-26T16:07:53.712Z] E                 namespace: odf-storage
[2024-09-26T16:07:53.712Z] E                 ownerReferences:
[2024-09-26T16:07:53.712Z] E                 - apiVersion: batch/v1
[2024-09-26T16:07:53.712Z] E                   blockOwnerDeletion: true
[2024-09-26T16:07:53.712Z] E                   controller: true
[2024-09-26T16:07:53.712Z] E                   kind: Job
[2024-09-26T16:07:53.712Z] E                   name: rook-ceph-osd-prepare-9afb3985e0230a85d5dd03e691b9c20d
[2024-09-26T16:07:53.712Z] E                   uid: 853f612d-fc0e-48d8-b2bb-735be68a74a0
[2024-09-26T16:07:53.712Z] E                 resourceVersion: '27373'
[2024-09-26T16:07:53.712Z] E                 uid: acec8b4b-784c-494c-a779-033be1402815
[2024-09-26T16:07:53.712Z] E               spec:
[2024-09-26T16:07:53.712Z] E                 affinity:
[2024-09-26T16:07:53.712Z] E                   nodeAffinity:
[2024-09-26T16:07:53.712Z] E                     requiredDuringSchedulingIgnoredDuringExecution:
[2024-09-26T16:07:53.712Z] E                       nodeSelectorTerms:
[2024-09-26T16:07:53.712Z] E                       - matchExpressions:
[2024-09-26T16:07:53.712Z] E                         - key: cluster.ocs.openshift.io/openshift-storage
[2024-09-26T16:07:53.712Z] E                           operator: Exists
[2024-09-26T16:07:53.712Z] E                         - key: cluster.ocs.openshift.io/openshift-storage
[2024-09-26T16:07:53.712Z] E                           operator: Exists
[2024-09-26T16:07:53.712Z] E                 containers:
[2024-09-26T16:07:53.712Z] E                 - args:
[2024-09-26T16:07:53.712Z] E                   - ceph
[2024-09-26T16:07:53.712Z] E                   - osd
[2024-09-26T16:07:53.712Z] E                   - provision
[2024-09-26T16:07:53.712Z] E                   command:
[2024-09-26T16:07:53.712Z] E                   - /rook/rook
[2024-09-26T16:07:53.712Z] E                   env:
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_NODE_NAME
[2024-09-26T16:07:53.712Z] E                     value: ocs-deviceset-gp3-csi-0-data-0l529m
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_CLUSTER_ID
[2024-09-26T16:07:53.712Z] E                     value: 8a0dd8af-c269-4580-bf85-d6f5a9e69fac
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_CLUSTER_NAME
[2024-09-26T16:07:53.712Z] E                     value: ocs-storagecluster-cephcluster
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_PRIVATE_IP
[2024-09-26T16:07:53.712Z] E                     valueFrom:
[2024-09-26T16:07:53.712Z] E                       fieldRef:
[2024-09-26T16:07:53.712Z] E                         apiVersion: v1
[2024-09-26T16:07:53.712Z] E                         fieldPath: status.podIP
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_PUBLIC_IP
[2024-09-26T16:07:53.712Z] E                     valueFrom:
[2024-09-26T16:07:53.712Z] E                       fieldRef:
[2024-09-26T16:07:53.712Z] E                         apiVersion: v1
[2024-09-26T16:07:53.712Z] E                         fieldPath: status.podIP
[2024-09-26T16:07:53.712Z] E                   - name: POD_NAMESPACE
[2024-09-26T16:07:53.712Z] E                     value: odf-storage
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_MON_ENDPOINTS
[2024-09-26T16:07:53.712Z] E                     valueFrom:
[2024-09-26T16:07:53.712Z] E                       configMapKeyRef:
[2024-09-26T16:07:53.712Z] E                         key: data
[2024-09-26T16:07:53.712Z] E                         name: rook-ceph-mon-endpoints
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_CONFIG_DIR
[2024-09-26T16:07:53.712Z] E                     value: /var/lib/rook
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_CEPH_CONFIG_OVERRIDE
[2024-09-26T16:07:53.712Z] E                     value: /etc/rook/config/override.conf
[2024-09-26T16:07:53.712Z] E                   - name: NODE_NAME
[2024-09-26T16:07:53.712Z] E                     valueFrom:
[2024-09-26T16:07:53.712Z] E                       fieldRef:
[2024-09-26T16:07:53.712Z] E                         apiVersion: v1
[2024-09-26T16:07:53.712Z] E                         fieldPath: spec.nodeName
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_CRUSHMAP_ROOT
[2024-09-26T16:07:53.712Z] E                     value: default
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_CEPH_USERNAME
[2024-09-26T16:07:53.712Z] E                     valueFrom:
[2024-09-26T16:07:53.712Z] E                       secretKeyRef:
[2024-09-26T16:07:53.712Z] E                         key: ceph-username
[2024-09-26T16:07:53.712Z] E                         name: rook-ceph-mon
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_FSID
[2024-09-26T16:07:53.712Z] E                     valueFrom:
[2024-09-26T16:07:53.712Z] E                       secretKeyRef:
[2024-09-26T16:07:53.712Z] E                         key: fsid
[2024-09-26T16:07:53.712Z] E                         name: rook-ceph-mon
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_OSD_STORE_TYPE
[2024-09-26T16:07:53.712Z] E                     value: bluestore
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_CRUSHMAP_HOSTNAME
[2024-09-26T16:07:53.712Z] E                     value: ocs-deviceset-gp3-csi-0-data-0l529m
[2024-09-26T16:07:53.712Z] E                   - name: CEPH_VOLUME_DEBUG
[2024-09-26T16:07:53.712Z] E                     value: '1'
[2024-09-26T16:07:53.712Z] E                   - name: CEPH_VOLUME_SKIP_RESTORECON
[2024-09-26T16:07:53.712Z] E                     value: '1'
[2024-09-26T16:07:53.712Z] E                   - name: DM_DISABLE_UDEV
[2024-09-26T16:07:53.712Z] E                     value: '1'
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_LOG_LEVEL
[2024-09-26T16:07:53.712Z] E                     value: DEBUG
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_CEPH_VERSION
[2024-09-26T16:07:53.712Z] E                     value: ceph version 18.2.1-229 reef
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_OSD_CRUSH_DEVICE_CLASS
[2024-09-26T16:07:53.712Z] E                     value: ssd
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_OSD_CRUSH_INITIAL_WEIGHT
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_DATA_DEVICES
[2024-09-26T16:07:53.712Z] E                     value: '[{"id":"/mnt/ocs-deviceset-gp3-csi-0-data-0l529m","storeConfig":{"osdsPerDevice":1}}]'
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_PVC_BACKED_OSD
[2024-09-26T16:07:53.712Z] E                     value: 'true'
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_ENCRYPTED_DEVICE
[2024-09-26T16:07:53.712Z] E                     value: 'true'
[2024-09-26T16:07:53.712Z] E                   - name: ROOK_PVC_NAME
[2024-09-26T16:07:53.712Z] E                     value: ocs-deviceset-gp3-csi-0-data-0l529m
[2024-09-26T16:07:53.712Z] E                   - name: CEPH_VOLUME_DMCRYPT_SECRET
[2024-09-26T16:07:53.712Z] E                     valueFrom:
[2024-09-26T16:07:53.712Z] E                       secretKeyRef:
[2024-09-26T16:07:53.712Z] E                         key: dmcrypt-key
[2024-09-26T16:07:53.712Z] E                         name: rook-ceph-osd-encryption-key-ocs-deviceset-gp3-csi-0-data-0l529m
[2024-09-26T16:07:53.712Z] E                   envFrom:
[2024-09-26T16:07:53.712Z] E                   - configMapRef:
[2024-09-26T16:07:53.712Z] E                       name: rook-ceph-osd-env-override
[2024-09-26T16:07:53.712Z] E                       optional: true
[2024-09-26T16:07:53.712Z] E                   image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
[2024-09-26T16:07:53.712Z] E                   imagePullPolicy: IfNotPresent
[2024-09-26T16:07:53.712Z] E                   name: provision
[2024-09-26T16:07:53.712Z] E                   resources: {}
[2024-09-26T16:07:53.712Z] E                   securityContext:
[2024-09-26T16:07:53.712Z] E                     capabilities:
[2024-09-26T16:07:53.712Z] E                       drop:
[2024-09-26T16:07:53.712Z] E                       - ALL
[2024-09-26T16:07:53.712Z] E                     privileged: true
[2024-09-26T16:07:53.712Z] E                     readOnlyRootFilesystem: false
[2024-09-26T16:07:53.712Z] E                     runAsNonRoot: false
[2024-09-26T16:07:53.712Z] E                     runAsUser: 0
[2024-09-26T16:07:53.712Z] E                   terminationMessagePath: /dev/termination-log
[2024-09-26T16:07:53.712Z] E                   terminationMessagePolicy: File
[2024-09-26T16:07:53.712Z] E                   volumeMounts:
[2024-09-26T16:07:53.712Z] E                   - mountPath: /var/lib/rook
[2024-09-26T16:07:53.712Z] E                     name: rook-data
[2024-09-26T16:07:53.712Z] E                   - mountPath: /etc/ceph
[2024-09-26T16:07:53.712Z] E                     name: ceph-conf-emptydir
[2024-09-26T16:07:53.712Z] E                   - mountPath: /run/ceph
[2024-09-26T16:07:53.712Z] E                     name: ceph-daemons-sock-dir
[2024-09-26T16:07:53.712Z] E                   - mountPath: /var/log/ceph
[2024-09-26T16:07:53.712Z] E                     name: rook-ceph-log
[2024-09-26T16:07:53.712Z] E                   - mountPath: /var/lib/ceph/crash
[2024-09-26T16:07:53.712Z] E                     name: rook-ceph-crash
[2024-09-26T16:07:53.712Z] E                   - mountPath: /dev
[2024-09-26T16:07:53.712Z] E                     name: devices
[2024-09-26T16:07:53.712Z] E                   - mountPath: /run/udev
[2024-09-26T16:07:53.712Z] E                     name: udev
[2024-09-26T16:07:53.712Z] E                   - mountPath: /rook
[2024-09-26T16:07:53.712Z] E                     name: rook-binaries
[2024-09-26T16:07:53.712Z] E                   - mountPath: /var/lib/rook-ceph-mon
[2024-09-26T16:07:53.712Z] E                     name: ceph-admin-secret
[2024-09-26T16:07:53.712Z] E                     readOnly: true
[2024-09-26T16:07:53.712Z] E                   - mountPath: /mnt
[2024-09-26T16:07:53.712Z] E                     name: ocs-deviceset-gp3-csi-0-data-0l529m-bridge
[2024-09-26T16:07:53.712Z] E                   - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
[2024-09-26T16:07:53.712Z] E                     name: kube-api-access-tv46g
[2024-09-26T16:07:53.712Z] E                     readOnly: true
[2024-09-26T16:07:53.712Z] E                 dnsPolicy: ClusterFirst
[2024-09-26T16:07:53.712Z] E                 enableServiceLinks: true
[2024-09-26T16:07:53.712Z] E                 hostIPC: true
[2024-09-26T16:07:53.712Z] E                 imagePullSecrets:
[2024-09-26T16:07:53.712Z] E                 - name: rook-ceph-osd-dockercfg-z9xhw
[2024-09-26T16:07:53.712Z] E                 initContainers:
[2024-09-26T16:07:53.712Z] E                 - args:
[2024-09-26T16:07:53.712Z] E                   - --archive
[2024-09-26T16:07:53.712Z] E                   - --force
[2024-09-26T16:07:53.712Z] E                   - --verbose
[2024-09-26T16:07:53.712Z] E                   - /usr/local/bin/rook
[2024-09-26T16:07:53.712Z] E                   - /rook
[2024-09-26T16:07:53.712Z] E                   command:
[2024-09-26T16:07:53.712Z] E                   - cp
[2024-09-26T16:07:53.712Z] E                   image: registry.redhat.io/odf4/rook-ceph-rhel9-operator@sha256:7d7b604c6a16d617c131cb7352f049ed15681f06e8d558658fd22c47a0ee717c
[2024-09-26T16:07:53.712Z] E                   imagePullPolicy: IfNotPresent
[2024-09-26T16:07:53.712Z] E                   name: copy-bins
[2024-09-26T16:07:53.712Z] E                   resources: {}
[2024-09-26T16:07:53.712Z] E                   securityContext:
[2024-09-26T16:07:53.712Z] E                     capabilities:
[2024-09-26T16:07:53.712Z] E                       drop:
[2024-09-26T16:07:53.712Z] E                       - ALL
[2024-09-26T16:07:53.712Z] E                   terminationMessagePath: /dev/termination-log
[2024-09-26T16:07:53.712Z] E                   terminationMessagePolicy: File
[2024-09-26T16:07:53.712Z] E                   volumeMounts:
[2024-09-26T16:07:53.712Z] E                   - mountPath: /rook
[2024-09-26T16:07:53.712Z] E                     name: rook-binaries
[2024-09-26T16:07:53.712Z] E                   - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
[2024-09-26T16:07:53.712Z] E                     name: kube-api-access-tv46g
[2024-09-26T16:07:53.712Z] E                     readOnly: true
[2024-09-26T16:07:53.712Z] E                 - command:
[2024-09-26T16:07:53.712Z] E                   - /bin/bash
[2024-09-26T16:07:53.712Z] E                   - -c
[2024-09-26T16:07:53.712Z] E                   - "\nset -xe\n\nPVC_SOURCE=/ocs-deviceset-gp3-csi-0-data-0l529m\nPVC_DEST=/mnt/ocs-deviceset-gp3-csi-0-data-0l529m\n\
[2024-09-26T16:07:53.712Z] E                     CP_ARGS=(--archive --dereference --verbose)\n\nif [ -b \"$PVC_DEST\" ];\
[2024-09-26T16:07:53.712Z] E                     \ then\n\tPVC_SOURCE_MAJ_MIN=$(stat --format '%t%T' $PVC_SOURCE)\n\tPVC_DEST_MAJ_MIN=$(stat\
[2024-09-26T16:07:53.712Z] E                     \ --format '%t%T' $PVC_DEST)\n\tif [[ \"$PVC_SOURCE_MAJ_MIN\" == \"$PVC_DEST_MAJ_MIN\"\
[2024-09-26T16:07:53.712Z] E                     \ ]]; then\n\t\techo \"PVC $PVC_DEST already exists and has the same major\
[2024-09-26T16:07:53.712Z] E                     \ and minor as $PVC_SOURCE: \"$PVC_SOURCE_MAJ_MIN\"\"\n\t\texit 0\n\telse\n\
[2024-09-26T16:07:53.712Z] E                     \t\techo \"PVC's source major/minor numbers changed\"\n\t\tCP_ARGS+=(--remove-destination)\n\
[2024-09-26T16:07:53.712Z] E                     \tfi\nfi\n\ncp \"${CP_ARGS[@]}\" \"$PVC_SOURCE\" \"$PVC_DEST\"\n"
[2024-09-26T16:07:53.712Z] E                   image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
[2024-09-26T16:07:53.712Z] E                   imagePullPolicy: IfNotPresent
[2024-09-26T16:07:53.712Z] E                   name: blkdevmapper
[2024-09-26T16:07:53.712Z] E                   resources: {}
[2024-09-26T16:07:53.712Z] E                   securityContext:
[2024-09-26T16:07:53.712Z] E                     capabilities:
[2024-09-26T16:07:53.712Z] E                       add:
[2024-09-26T16:07:53.712Z] E                       - MKNOD
[2024-09-26T16:07:53.712Z] E                       drop:
[2024-09-26T16:07:53.712Z] E                       - ALL
[2024-09-26T16:07:53.712Z] E                     privileged: true
[2024-09-26T16:07:53.712Z] E                   terminationMessagePath: /dev/termination-log
[2024-09-26T16:07:53.712Z] E                   terminationMessagePolicy: File
[2024-09-26T16:07:53.712Z] E                   volumeDevices:
[2024-09-26T16:07:53.712Z] E                   - devicePath: /ocs-deviceset-gp3-csi-0-data-0l529m
[2024-09-26T16:07:53.712Z] E                     name: ocs-deviceset-gp3-csi-0-data-0l529m
[2024-09-26T16:07:53.712Z] E                   volumeMounts:
[2024-09-26T16:07:53.712Z] E                   - mountPath: /mnt
[2024-09-26T16:07:53.712Z] E                     name: ocs-deviceset-gp3-csi-0-data-0l529m-bridge
[2024-09-26T16:07:53.712Z] E                   - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
[2024-09-26T16:07:53.712Z] E                     name: kube-api-access-tv46g
[2024-09-26T16:07:53.712Z] E                     readOnly: true
[2024-09-26T16:07:53.712Z] E                 nodeName: ip-10-0-0-122.us-west-2.compute.internal
[2024-09-26T16:07:53.712Z] E                 preemptionPolicy: PreemptLowerPriority
[2024-09-26T16:07:53.712Z] E                 priority: 2000001000
[2024-09-26T16:07:53.712Z] E                 priorityClassName: system-node-critical
[2024-09-26T16:07:53.712Z] E                 restartPolicy: OnFailure
[2024-09-26T16:07:53.712Z] E                 schedulerName: default-scheduler
[2024-09-26T16:07:53.712Z] E                 securityContext:
[2024-09-26T16:07:53.712Z] E                   fsGroup: 1000790000
[2024-09-26T16:07:53.712Z] E                   seLinuxOptions:
[2024-09-26T16:07:53.712Z] E                     level: s0:c28,c17
[2024-09-26T16:07:53.712Z] E                 serviceAccount: rook-ceph-osd
[2024-09-26T16:07:53.713Z] E                 serviceAccountName: rook-ceph-osd
[2024-09-26T16:07:53.713Z] E                 terminationGracePeriodSeconds: 30
[2024-09-26T16:07:53.713Z] E                 tolerations:
[2024-09-26T16:07:53.713Z] E                 - effect: NoSchedule
[2024-09-26T16:07:53.713Z] E                   key: node.ocs.openshift.io/storage
[2024-09-26T16:07:53.713Z] E                   operator: Equal
[2024-09-26T16:07:53.713Z] E                   value: 'true'
[2024-09-26T16:07:53.713Z] E                 - effect: NoSchedule
[2024-09-26T16:07:53.713Z] E                   key: node.ocs.openshift.io/storage
[2024-09-26T16:07:53.713Z] E                   operator: Equal
[2024-09-26T16:07:53.713Z] E                   value: 'true'
[2024-09-26T16:07:53.713Z] E                 - effect: NoExecute
[2024-09-26T16:07:53.713Z] E                   key: node.kubernetes.io/not-ready
[2024-09-26T16:07:53.713Z] E                   operator: Exists
[2024-09-26T16:07:53.713Z] E                   tolerationSeconds: 300
[2024-09-26T16:07:53.713Z] E                 - effect: NoExecute
[2024-09-26T16:07:53.713Z] E                   key: node.kubernetes.io/unreachable
[2024-09-26T16:07:53.713Z] E                   operator: Exists
[2024-09-26T16:07:53.713Z] E                   tolerationSeconds: 300
[2024-09-26T16:07:53.713Z] E                 topologySpreadConstraints:
[2024-09-26T16:07:53.713Z] E                 - labelSelector:
[2024-09-26T16:07:53.713Z] E                     matchExpressions:
[2024-09-26T16:07:53.713Z] E                     - key: app
[2024-09-26T16:07:53.713Z] E                       operator: In
[2024-09-26T16:07:53.713Z] E                       values:
[2024-09-26T16:07:53.713Z] E                       - rook-ceph-osd
[2024-09-26T16:07:53.713Z] E                       - rook-ceph-osd-prepare
[2024-09-26T16:07:53.713Z] E                   maxSkew: 1
[2024-09-26T16:07:53.713Z] E                   topologyKey: topology.rook.io/rack
[2024-09-26T16:07:53.713Z] E                   whenUnsatisfiable: DoNotSchedule
[2024-09-26T16:07:53.713Z] E                 - labelSelector:
[2024-09-26T16:07:53.713Z] E                     matchExpressions:
[2024-09-26T16:07:53.713Z] E                     - key: app
[2024-09-26T16:07:53.713Z] E                       operator: In
[2024-09-26T16:07:53.713Z] E                       values:
[2024-09-26T16:07:53.713Z] E                       - rook-ceph-osd
[2024-09-26T16:07:53.713Z] E                       - rook-ceph-osd-prepare
[2024-09-26T16:07:53.713Z] E                   maxSkew: 1
[2024-09-26T16:07:53.713Z] E                   topologyKey: kubernetes.io/hostname
[2024-09-26T16:07:53.713Z] E                   whenUnsatisfiable: ScheduleAnyway
[2024-09-26T16:07:53.713Z] E                 volumes:
[2024-09-26T16:07:53.713Z] E                 - hostPath:
[2024-09-26T16:07:53.713Z] E                     path: /var/lib/rook
[2024-09-26T16:07:53.713Z] E                     type: ''
[2024-09-26T16:07:53.713Z] E                   name: rook-data
[2024-09-26T16:07:53.713Z] E                 - emptyDir: {}
[2024-09-26T16:07:53.713Z] E                   name: ceph-conf-emptydir
[2024-09-26T16:07:53.713Z] E                 - hostPath:
[2024-09-26T16:07:53.713Z] E                     path: /var/lib/rook/exporter
[2024-09-26T16:07:53.713Z] E                     type: DirectoryOrCreate
[2024-09-26T16:07:53.713Z] E                   name: ceph-daemons-sock-dir
[2024-09-26T16:07:53.713Z] E                 - hostPath:
[2024-09-26T16:07:53.713Z] E                     path: /var/lib/rook/odf-storage/log
[2024-09-26T16:07:53.713Z] E                     type: ''
[2024-09-26T16:07:53.713Z] E                   name: rook-ceph-log
[2024-09-26T16:07:53.713Z] E                 - hostPath:
[2024-09-26T16:07:53.713Z] E                     path: /var/lib/rook/odf-storage/crash
[2024-09-26T16:07:53.713Z] E                     type: ''
[2024-09-26T16:07:53.713Z] E                   name: rook-ceph-crash
[2024-09-26T16:07:53.713Z] E                 - emptyDir: {}
[2024-09-26T16:07:53.713Z] E                   name: rook-binaries
[2024-09-26T16:07:53.713Z] E                 - hostPath:
[2024-09-26T16:07:53.713Z] E                     path: /run/udev
[2024-09-26T16:07:53.713Z] E                     type: ''
[2024-09-26T16:07:53.713Z] E                   name: udev
[2024-09-26T16:07:53.713Z] E                 - hostPath:
[2024-09-26T16:07:53.713Z] E                     path: /dev
[2024-09-26T16:07:53.713Z] E                     type: ''
[2024-09-26T16:07:53.713Z] E                   name: devices
[2024-09-26T16:07:53.713Z] E                 - name: ceph-admin-secret
[2024-09-26T16:07:53.713Z] E                   secret:
[2024-09-26T16:07:53.713Z] E                     defaultMode: 420
[2024-09-26T16:07:53.713Z] E                     items:
[2024-09-26T16:07:53.713Z] E                     - key: ceph-secret
[2024-09-26T16:07:53.713Z] E                       path: secret.keyring
[2024-09-26T16:07:53.713Z] E                     secretName: rook-ceph-mon
[2024-09-26T16:07:53.713Z] E                 - name: ocs-deviceset-gp3-csi-0-data-0l529m
[2024-09-26T16:07:53.713Z] E                   persistentVolumeClaim:
[2024-09-26T16:07:53.713Z] E                     claimName: ocs-deviceset-gp3-csi-0-data-0l529m
[2024-09-26T16:07:53.713Z] E                 - emptyDir:
[2024-09-26T16:07:53.713Z] E                     medium: Memory
[2024-09-26T16:07:53.713Z] E                   name: ocs-deviceset-gp3-csi-0-data-0l529m-bridge
[2024-09-26T16:07:53.713Z] E                 - name: kube-api-access-tv46g
[2024-09-26T16:07:53.713Z] E                   projected:
[2024-09-26T16:07:53.713Z] E                     defaultMode: 420
[2024-09-26T16:07:53.713Z] E                     sources:
[2024-09-26T16:07:53.713Z] E                     - serviceAccountToken:
[2024-09-26T16:07:53.713Z] E                         expirationSeconds: 3607
[2024-09-26T16:07:53.713Z] E                         path: token
[2024-09-26T16:07:53.713Z] E                     - configMap:
[2024-09-26T16:07:53.713Z] E                         items:
[2024-09-26T16:07:53.713Z] E                         - key: ca.crt
[2024-09-26T16:07:53.713Z] E                           path: ca.crt
[2024-09-26T16:07:53.713Z] E                         name: kube-root-ca.crt
[2024-09-26T16:07:53.713Z] E                     - downwardAPI:
[2024-09-26T16:07:53.713Z] E                         items:
[2024-09-26T16:07:53.713Z] E                         - fieldRef:
[2024-09-26T16:07:53.713Z] E                             apiVersion: v1
[2024-09-26T16:07:53.713Z] E                             fieldPath: metadata.namespace
[2024-09-26T16:07:53.713Z] E                           path: namespace
[2024-09-26T16:07:53.713Z] E                     - configMap:
[2024-09-26T16:07:53.713Z] E                         items:
[2024-09-26T16:07:53.713Z] E                         - key: service-ca.crt
[2024-09-26T16:07:53.713Z] E                           path: service-ca.crt
[2024-09-26T16:07:53.713Z] E                         name: openshift-service-ca.crt
[2024-09-26T16:07:53.713Z] E               status:
[2024-09-26T16:07:53.713Z] E                 conditions:
[2024-09-26T16:07:53.713Z] E                 - lastProbeTime: null
[2024-09-26T16:07:53.713Z] E                   lastTransitionTime: '2024-09-26T07:42:09Z'
[2024-09-26T16:07:53.713Z] E                   status: 'False'
[2024-09-26T16:07:53.713Z] E                   type: PodReadyToStartContainers
[2024-09-26T16:07:53.713Z] E                 - lastProbeTime: null
[2024-09-26T16:07:53.713Z] E                   lastTransitionTime: '2024-09-26T07:41:53Z'
[2024-09-26T16:07:53.713Z] E                   reason: PodCompleted
[2024-09-26T16:07:53.713Z] E                   status: 'True'
[2024-09-26T16:07:53.713Z] E                   type: Initialized
[2024-09-26T16:07:53.713Z] E                 - lastProbeTime: null
[2024-09-26T16:07:53.713Z] E                   lastTransitionTime: '2024-09-26T07:42:08Z'
[2024-09-26T16:07:53.713Z] E                   reason: PodCompleted
[2024-09-26T16:07:53.713Z] E                   status: 'False'
[2024-09-26T16:07:53.713Z] E                   type: Ready
[2024-09-26T16:07:53.713Z] E                 - lastProbeTime: null
[2024-09-26T16:07:53.713Z] E                   lastTransitionTime: '2024-09-26T07:42:08Z'
[2024-09-26T16:07:53.713Z] E                   reason: PodCompleted
[2024-09-26T16:07:53.713Z] E                   status: 'False'
[2024-09-26T16:07:53.713Z] E                   type: ContainersReady
[2024-09-26T16:07:53.713Z] E                 - lastProbeTime: null
[2024-09-26T16:07:53.713Z] E                   lastTransitionTime: '2024-09-26T07:41:46Z'
[2024-09-26T16:07:53.713Z] E                   status: 'True'
[2024-09-26T16:07:53.713Z] E                   type: PodScheduled
[2024-09-26T16:07:53.713Z] E                 containerStatuses:
[2024-09-26T16:07:53.713Z] E                 - containerID: cri-o://3e2708252a6a42c36825a725954ea34d91abab861989b6a60d9409c2384ac409
[2024-09-26T16:07:53.713Z] E                   image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
[2024-09-26T16:07:53.713Z] E                   imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad
[2024-09-26T16:07:53.713Z] E                   lastState: {}
[2024-09-26T16:07:53.713Z] E                   name: provision
[2024-09-26T16:07:53.713Z] E                   ready: false
[2024-09-26T16:07:53.713Z] E                   restartCount: 0
[2024-09-26T16:07:53.713Z] E                   started: false
[2024-09-26T16:07:53.713Z] E                   state:
[2024-09-26T16:07:53.713Z] E                     terminated:
[2024-09-26T16:07:53.713Z] E                       containerID: cri-o://3e2708252a6a42c36825a725954ea34d91abab861989b6a60d9409c2384ac409
[2024-09-26T16:07:53.713Z] E                       exitCode: 0
[2024-09-26T16:07:53.713Z] E                       finishedAt: '2024-09-26T07:42:08Z'
[2024-09-26T16:07:53.713Z] E                       reason: Completed
[2024-09-26T16:07:53.713Z] E                       startedAt: '2024-09-26T07:41:53Z'
[2024-09-26T16:07:53.713Z] E                 hostIP: 10.0.0.122
[2024-09-26T16:07:53.713Z] E                 hostIPs:
[2024-09-26T16:07:53.713Z] E                 - ip: 10.0.0.122
[2024-09-26T16:07:53.713Z] E                 initContainerStatuses:
[2024-09-26T16:07:53.713Z] E                 - containerID: cri-o://46f7e06292b0cef690881ab8748f5a7fa245ab1550c2d4555e10815c9949dc48
[2024-09-26T16:07:53.713Z] E                   image: registry.redhat.io/odf4/rook-ceph-rhel9-operator@sha256:7d7b604c6a16d617c131cb7352f049ed15681f06e8d558658fd22c47a0ee717c
[2024-09-26T16:07:53.713Z] E                   imageID: registry.redhat.io/odf4/rook-ceph-rhel9-operator@sha256:7d7b604c6a16d617c131cb7352f049ed15681f06e8d558658fd22c47a0ee717c
[2024-09-26T16:07:53.713Z] E                   lastState: {}
[2024-09-26T16:07:53.713Z] E                   name: copy-bins
[2024-09-26T16:07:53.713Z] E                   ready: true
[2024-09-26T16:07:53.713Z] E                   restartCount: 0
[2024-09-26T16:07:53.713Z] E                   started: false
[2024-09-26T16:07:53.713Z] E                   state:
[2024-09-26T16:07:53.713Z] E                     terminated:
[2024-09-26T16:07:53.713Z] E                       containerID: cri-o://46f7e06292b0cef690881ab8748f5a7fa245ab1550c2d4555e10815c9949dc48
[2024-09-26T16:07:53.713Z] E                       exitCode: 0
[2024-09-26T16:07:53.713Z] E                       finishedAt: '2024-09-26T07:41:52Z'
[2024-09-26T16:07:53.713Z] E                       reason: Completed
[2024-09-26T16:07:53.713Z] E                       startedAt: '2024-09-26T07:41:52Z'
[2024-09-26T16:07:53.713Z] E                 - containerID: cri-o://5de1b69405ea8ba11ec666670d2e0822d46a9af8eee29473d61abd2353c2684a
[2024-09-26T16:07:53.713Z] E                   image: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:75bd8969ab3f86f2203a1ceb187876f44e54c9ee3b917518c4d696cf6cd88ce3
[2024-09-26T16:07:53.713Z] E                   imageID: registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:4f598dcdef399669e615b5624fd2ff3c4d152e44da2614e5aa5e286d628158ad
[2024-09-26T16:07:53.713Z] E                   lastState: {}
[2024-09-26T16:07:53.713Z] E                   name: blkdevmapper
[2024-09-26T16:07:53.713Z] E                   ready: true
[2024-09-26T16:07:53.713Z] E                   restartCount: 0
[2024-09-26T16:07:53.713Z] E                   started: false
[2024-09-26T16:07:53.713Z] E                   state:
[2024-09-26T16:07:53.713Z] E                     terminated:
[2024-09-26T16:07:53.713Z] E                       containerID: cri-o://5de1b69405ea8ba11ec666670d2e0822d46a9af8eee29473d61abd2353c2684a
[2024-09-26T16:07:53.713Z] E                       exitCode: 0
[2024-09-26T16:07:53.713Z] E                       finishedAt: '2024-09-26T07:41:52Z'
[2024-09-26T16:07:53.713Z] E                       reason: Completed
[2024-09-26T16:07:53.713Z] E                       startedAt: '2024-09-26T07:41:52Z'
[2024-09-26T16:07:53.713Z] E                 phase: Succeeded
[2024-09-26T16:07:53.713Z] E                 podIP: 10.129.0.45
[2024-09-26T16:07:53.713Z] E                 podIPs:
[2024-09-26T16:07:53.713Z] E                 - ip: 10.129.0.45
[2024-09-26T16:07:53.713Z] E                 qosClass: BestEffort
[2024-09-26T16:07:53.713Z] E                 startTime: '2024-09-26T07:41:46Z'
[2024-09-26T16:07:53.713Z] 
[2024-09-26T16:07:53.713Z] /home/jenkins/workspace/qe-deploy-ocs-cluster/ocs-ci/ocs_ci/utility/environment_check.py:269: ResourceLeftoversException
DanielOsypenko commented 1 month ago

hello @nagendra202, tagging you to check if this test should pass the tierDown as is, or we need to make changes to the test. The test has been recently added by you. Thanks

nagendra202 commented 1 month ago

Ack.

Not sure, why the pod "name: rook-ceph-osd-prepare-9afb3985e0230a85d5dd03e691b9c20d-bjkkn" is being created. Need to fix it.