red-hat-storage / ocs-ci

https://ocs-ci.readthedocs.io/en/latest/
MIT License
108 stars 166 forks source link

test_pvc_expansion_when_full leftover PV error #10740

Open jilju opened 1 week ago

jilju commented 1 week ago

The test case test_pvc_expansion_when_full is reporting error due to leftover PV

      raise exceptions.ResourceLeftoversException(
            f"\nThere are leftovers in the environment after test case:"
            f"\nResources added:\n{yaml.dump(leftovers['Leftovers added'])}"
            f"\nResources "
            f"removed:\n {yaml.dump(leftovers['Leftovers removed'])}"
        )
E           ocs_ci.ocs.exceptions.ResourceLeftoversException:
E           There are leftovers in the environment after test case:
E           Resources added:
E           []
E
E           Resources removed:
E            - 'pvs':
E             - apiVersion: v1
E               kind: PersistentVolume
E               metadata:
E                 annotations:
E                   pv.kubernetes.io/provisioned-by: openshift-storage.rbd.csi.ceph.com
E                   volume.kubernetes.io/provisioner-deletion-secret-name: rook-csi-rbd-provisioner
E                   volume.kubernetes.io/provisioner-deletion-secret-namespace: openshift-storage
E                 creationTimestamp: '2024-10-22T08:43:38Z'
E                 finalizers:
E                 - kubernetes.io/pv-protection
E                 labels:
E                   fixture: workload_storageutilization_checksum_rbd
E                 name: pvc-e1625b34-c8ef-49bf-bfda-d5939c37bedc
E                 resourceVersion: '125478'
E                 uid: 371e2d49-95f0-4385-b24c-226af7012679
E               spec:
E                 accessModes:
E                 - ReadWriteOnce
E                 capacity:
E                   storage: 10Gi
E                 csi:
E                   controllerExpandSecretRef:
E                     name: rook-csi-rbd-provisioner
E                     namespace: openshift-storage
E                   driver: openshift-storage.rbd.csi.ceph.com
E                   fsType: ext4
E                   nodeStageSecretRef:
E                     name: rook-csi-rbd-node
E                     namespace: openshift-storage
E                   volumeAttributes:
E                     clusterID: openshift-storage
E                     imageFeatures: layering,deep-flatten,exclusive-lock,object-map,fast-diff
E                     imageFormat: '2'
E                     imageName: csi-vol-49f437cc-4558-41c5-a5b5-4a345e6bef6a
E                     journalPool: rbd
E                     pool: rbd
E                     storage.kubernetes.io/csiProvisionerIdentity: 1729585766794-5657-openshift-storage.rbd.csi.ceph.com
E                   volumeHandle: 0001-0011-openshift-storage-0000000000000002-49f437cc-4558-41c5-a5b5-4a345e6bef6a
E                 persistentVolumeReclaimPolicy: Retain
E                 storageClassName: ocs-external-storagecluster-ceph-rbd
E                 volumeMode: Filesystem
E               status:
E                 lastPhaseTransitionTime: '2024-10-22T10:07:15Z'
E                 phase: Available

ocs_ci/utility/environment_check.py:269: ResourceLeftoversException

Test case - tests/functional/pv/pvc_resize/test_pvc_expansion_when_full.py::TestPvcExpansionWhenFull::test_pvc_expansion_when_full

Failed runs

https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/678/26219/1283545/1283590/log https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#OCS/launches/678/26158/1280285/1280330/log https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#OCS/launches/678/26144/1279470/1279515/log

jilju commented 3 days ago

In https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/738/25809/1253739/1253742/log PV was not deleted.

def finalizer():
        """
        Delete the PVC
        """
        pv_objs = []

    # Get PV form PVC instances and delete PVCs
    for instance in instances:
        if not instance.is_deleted:
            pv_objs.append(instance.backed_pv_obj)
            instance.delete()
            instance.ocp.wait_for_delete(instance.name)

    # Wait for PVs to delete
    # If they have ReclaimPolicy set to Retain then change to Delete
    for pv_obj in pv_objs:
        if (
            pv_obj.data.get("spec", {}).get("persistentVolumeReclaimPolicy")
            == constants.RECLAIM_POLICY_RETAIN
            and pv_obj is not None
        ):
            helpers.wait_for_resource_state(pv_obj, constants.STATUS_RELEASED)
            patch_param = '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
            pv_obj.ocp.patch(resource_name=pv_obj.name, params=patch_param)

      pv_obj.ocp.wait_for_delete(resource_name=pv_obj.name, timeout=180)

tests/conftest.py:1193: 

self = <ocs_ci.ocs.ocp.OCP object at 0x7ff243f74190>
resource_name = 'pvc-8dc47357-84e0-4798-b652-dd01c26b6f9e', timeout = 180
sleep = 3

def wait_for_delete(self, resource_name="", timeout=60, sleep=3):
    """
    Wait for a resource to be deleted

    Args:
        resource_name (str): The name of the resource to wait
            for (e.g.my-pv1)
        timeout (int): Time in seconds to wait
        sleep (int): Sampling time in seconds

    Raises:
        CommandFailed: If failed to verify the resource deletion
        TimeoutError: If resource is not deleted within specified timeout

    Returns:
        bool: True in case resource deletion is successful

    """
    if config.ENV_DATA["platform"].lower() == constants.IBM_POWER_PLATFORM:
        timeout = 720
    start_time = time.time()
    while True:
        try:
            self.get(resource_name=resource_name)
        except CommandFailed as ex:
            if "NotFound" in str(ex):
                log.info(f"{self.kind} {resource_name} got deleted successfully")
                return True
            else:
                raise ex

        if timeout < (time.time() - start_time):
            describe_out = self.describe(resource_name=resource_name)
            msg = (
                f"Timeout when waiting for {resource_name} to delete. "
                f"Describe output: {describe_out}"
            )

          raise TimeoutError(msg)
E               TimeoutError: Timeout when waiting for pvc-8dc47357-84e0-4798-b652-dd01c26b6f9e to delete. Describe output: Name:            pvc-8dc47357-84e0-4798-b652-dd01c26b6f9e
E               Labels:          
E               Annotations:     pv.kubernetes.io/provisioned-by: openshift-storage.rbd.csi.ceph.com
E                                volume.kubernetes.io/provisioner-deletion-secret-name: rook-csi-rbd-provisioner
E                                volume.kubernetes.io/provisioner-deletion-secret-namespace: openshift-storage
E               Finalizers:      [external-provisioner.volume.kubernetes.io/finalizer kubernetes.io/pv-protection]
E               StorageClass:    ocs-storagecluster-ceph-rbd
E               Status:          Released
E               Claim:           namespace-test-d8383851f8c84755890dbc12b/pvc-test-807cdea32af9434bbce853d2ab0a152
E               Reclaim Policy:  Delete
E               Access Modes:    RWO
E               VolumeMode:      Filesystem
E               Capacity:        10Gi
E               Node Affinity:   
E               Message:
E               Source:
E                   Type:              CSI (a Container Storage Interface (CSI) volume source)
E                   Driver:            openshift-storage.rbd.csi.ceph.com
E                   FSType:            ext4
E                   VolumeHandle:      0001-0011-openshift-storage-0000000000000001-b3c3121c-dde2-492f-bfd4-ecad901a81b6
E                   ReadOnly:          false
E                   VolumeAttributes:      clusterID=openshift-storage
E                                          imageFeatures=layering,deep-flatten,exclusive-lock,object-map,fast-diff
E                                          imageFormat=2
E                                          imageName=csi-vol-b3c3121c-dde2-492f-bfd4-ecad901a81b6
E                                          journalPool=ocs-storagecluster-cephblockpool
E                                          pool=ocs-storagecluster-cephblockpool
E                                          storage.kubernetes.io/csiProvisionerIdentity=1728663376444-6141-openshift-storage.rbd.csi.ceph.com
E               Events:
E                 Type     Reason              Age                  From                                                                                                                Message
E                 ----     ------              ----                 ----                                                                                                                -------
E                 Warning  VolumeFailedDelete  17s (x9 over 2m51s)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-6d9b88b95f-79xtc_81fd3569-b643-4618-ac10-b2b4a94ce900  rpc error: code = Internal desc = rbd csi-vol-b3c3121c-dde2-492f-bfd4-ecad901a81b6 is still being used

ocs_ci/ocs/ocp.py:932: TimeoutError