red-hat-storage / ocs-ci

https://ocs-ci.readthedocs.io/en/latest/
MIT License
108 stars 166 forks source link

test_noobaa_db_backup_and_recovery is failing at create_pvc_snapshot and teardown #9480

Closed vkathole closed 2 months ago

vkathole commented 7 months ago

RP : https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/632/18992/923805/923809/log https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/632/19195/932340/932344/log

`self = <test_noobaa_db_backup_and_recovery.TestNoobaaBackupAndRecovery object at 0x7fca2c9f06a0> pvc_factory = <function pvc_factory_fixture..factory at 0x7fca285e71f0> pod_factory = <function pod_factory_fixture..factory at 0x7fca285e7310> snapshot_factory = <function snapshot_factory_fixture..factory at 0x7fca285e7430> bucket_factory = <function bucket_factory_fixture.._create_buckets at 0x7fca27b11550> rgw_bucket_factory = <function bucket_factory_fixture.._create_buckets at 0x7fca27b11dc0> noobaa_db_backup_and_recovery = <function noobaa_db_backup_and_recovery..factory at 0x7fca28dc1310>

@pytest.mark.polarion_id("OCS-2605") @pytest.mark.bugzilla("1924047") @skipif_ocs_version("<4.6") def test_noobaa_db_backup_and_recovery( self, pvc_factory, pod_factory, snapshot_factory, bucket_factory, rgw_bucket_factory, noobaa_db_backup_and_recovery, ): """ Test case to verify noobaa backup and recovery

1. Take snapshot db-noobaa-db-0 PVC and retore it to PVC
2. Scale down the statefulset noobaa-db
3. Get the yaml of the current PVC, db-noobaa-db-0 and
   change the parameter persistentVolumeReclaimPolicy to Retain for restored PVC
4. Delete both PVCs, the PV for the original claim db-noobaa-db-0 will be removed.
   The PV for claim db-noobaa-db-0-snapshot-restore will move to ‘Released’
5. Edit again restore PV and remove the claimRef section.
   The volume will transition to Available.
6. Edit the yaml db-noobaa-db-0.yaml and change the setting volumeName to restored PVC.
7. Scale up the stateful set again and the pod should be running

"""

noobaa_db_backup_and_recovery(snapshot_factory=snapshot_factory)

tests/cross_functional/kcs/test_noobaa_db_backup_and_recovery.py:71:

tests/cross_functional/conftest.py:350: in factory snap_obj = snapshot_factory( tests/conftest.py:4043: in factory snap_obj = pvc_obj.create_snapshot(snapshot_name=snapshot_name, wait=wait) ocs_ci/ocs/resources/pvc.py:261: in create_snapshot snapshot_obj = create_pvc_snapshot( ocs_ci/ocs/resources/pvc.py:454: in create_pvc_snapshot ocs_obj.ocp.wait_for_resource( ocs_ci/ocs/ocp.py:809: in wait_for_resource raise (ex) ocs_ci/ocs/ocp.py:700: in wait_for_resource for sample in TimeoutSampler(

self = <ocs_ci.utility.utils.TimeoutSampler object at 0x7fca2841f730>

def iter(self): if self.start_time is None: self.start_time = time.time() while True: self.last_sample_time = time.time() if self.timeout <= (self.last_sample_time - self.start_time):

      raise self.timeout_exc_cls(*self.timeout_exc_args)

E ocs_ci.ocs.exceptions.TimeoutExpiredError: Timed out after 60s running get("db-noobaa-db-pg-0-snapshot", True, None)

ocs_ci/utility/utils.py:1309: TimeoutExpiredError`

PrasadDesala commented 7 months ago

def finalizer():

Get the statefulset replica count

    sst_obj = OCP(
        kind=constants.STATEFULSET,
        namespace=config.ENV_DATA["cluster_namespace"],
    )
    noobaa_db_sst_obj = sst_obj.get(resource_name=constants.NOOBAA_DB_STATEFULSET)
    if noobaa_db_sst_obj["spec"]["replicas"] != 1:
        modify_statefulset_replica_count(
            statefulset_name=constants.NOOBAA_DB_STATEFULSET, replica_count=1
        ), f"Failed to scale up the statefulset {constants.NOOBAA_DB_STATEFULSET}"

try:

  restore_pvc_objs[0].delete()

E IndexError: list index out of range

tests/cross_functional/conftest.py:469: IndexError

https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/632/18992/923805/923810/log https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/632/18473/896460/896463/log