red-hat-storage / ocs-ci

https://ocs-ci.readthedocs.io/en/latest/
MIT License
109 stars 166 forks source link

test_pvc_multiple_snapshot_performance both RBD (512) and CephFS (100) in 4.15 are failing #9218

Closed ypersky1980 closed 6 months ago

ypersky1980 commented 8 months ago

Test case is failing - re-run the test and determine whether this is a product bug ( open a bz) or a test bug ( submit a pr with a fix)

https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/all/17989/883997/884051/log?item1Params=page.page%3D2

self = <test_pvc_multi_snapshot_performance.TestPvcMultiSnapshotPerformance object at 0x7fe6708cb6d0>

def setup(self): """ Setting up the test environment : Calculating the amount of storage which available for the test Creating namespace (project) for the test

"""
log.info("Setting up the test environment")

super(TestPvcMultiSnapshotPerformance, self).setup()

tests/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py:64:

ocs_ci/ocs/perftests.py:97: in setup self.get_osd_info() ocs_ci/ocs/perftests.py:229: in get_osd_info osd_info = ct_pod.exec_ceph_cmd(ceph_cmd="ceph osd df") ocs_ci/ocs/resources/pod.py:345: in exec_ceph_cmd out = self.exec_cmd_on_pod( ocs_ci/ocs/resources/pod.py:192: in exec_cmd_on_pod return self.ocp.exec_oc_cmd( ocs_ci/ocs/ocp.py:178: in exec_oc_cmd out = run_cmd( ocs_ci/utility/utils.py:484: in run_cmd completed_process = exec_cmd( ocs_ci/utility/utils.py:633: in exec_cmd completed_process = subprocess.run( /usr/lib64/python3.8/subprocess.py:495: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib64/python3.8/subprocess.py:1028: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib64/python3.8/subprocess.py:1869: in _communicate self._check_timeout(endtime, orig_timeout, stdout, stderr)

self = <subprocess.Popen object at 0x7fe5864521c0>, endtime = 185149.65396911 orig_timeout = 600, stdout_seq = [], stderr_seq = [] skip_check_and_raise = False

def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq, skip_check_and_raise=False): """Convenience for checking if a timeout has expired.""" if endtime is None: return if skip_check_and_raise or _time() > endtime:

  raise TimeoutExpired(
            self.args, orig_timeout,
            output=b''.join(stdout_seq) if stdout_seq else None,
            stderr=b''.join(stderr_seq) if stderr_seq else None)

E subprocess.TimeoutExpired: Command '['oc', '--kubeconfig', '/home/jenkins/current-cluster-dir/openshift-cluster-dir/auth/kubeconfig', '-n', 'openshift-storage', 'rsh', 'rook-ceph-tools-7997d9b857-g4kns', 'ceph', 'osd', 'df', '--format', 'json-pretty']' timed out after 600 seconds

/usr/lib64/python3.8/subprocess.py:1072: TimeoutExpired

2024-01-13 14:31:46

ypersky1980 commented 8 months ago

https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#ocs/launches/all/17989/883997/884052/log?item1Params=page.page%3D2

self = <test_pvc_multi_snapshot_performance.TestPvcMultiSnapshotPerformance object at 0x7fe675aef0d0>

def setup(self): """ Setting up the test environment : Calculating the amount of storage which available for the test Creating namespace (project) for the test

"""
log.info("Setting up the test environment")

super(TestPvcMultiSnapshotPerformance, self).setup()

tests/e2e/performance/csi_tests/test_pvc_multi_snapshot_performance.py:64:

ocs_ci/ocs/perftests.py:97: in setup self.get_osd_info() ocs_ci/ocs/perftests.py:229: in get_osd_info osd_info = ct_pod.exec_ceph_cmd(ceph_cmd="ceph osd df") ocs_ci/ocs/resources/pod.py:345: in exec_ceph_cmd out = self.exec_cmd_on_pod( ocs_ci/ocs/resources/pod.py:192: in exec_cmd_on_pod return self.ocp.exec_oc_cmd( ocs_ci/ocs/ocp.py:178: in exec_oc_cmd out = run_cmd( ocs_ci/utility/utils.py:484: in run_cmd completed_process = exec_cmd( ocs_ci/utility/utils.py:633: in exec_cmd completed_process = subprocess.run( /usr/lib64/python3.8/subprocess.py:495: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib64/python3.8/subprocess.py:1028: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib64/python3.8/subprocess.py:1869: in _communicate self._check_timeout(endtime, orig_timeout, stdout, stderr)

self = <subprocess.Popen object at 0x7fe6d2c6adf0>, endtime = 186695.877046423 orig_timeout = 600, stdout_seq = [], stderr_seq = [] skip_check_and_raise = False

def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq, skip_check_and_raise=False): """Convenience for checking if a timeout has expired.""" if endtime is None: return if skip_check_and_raise or _time() > endtime:

  raise TimeoutExpired(
            self.args, orig_timeout,
            output=b''.join(stdout_seq) if stdout_seq else None,
            stderr=b''.join(stderr_seq) if stderr_seq else None)

E subprocess.TimeoutExpired: Command '['oc', '--kubeconfig', '/home/jenkins/current-cluster-dir/openshift-cluster-dir/auth/kubeconfig', '-n', 'openshift-storage', 'rsh', 'rook-ceph-tools-7997d9b857-g4kns', 'ceph', 'osd', 'df', '--format', 'json-pretty']' timed out after 600 seconds

/usr/lib64/python3.8/subprocess.py:1072: TimeoutExpired

2024-01-13 14:42:53

ypersky1980 commented 7 months ago

https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/33847/

failed on teardown with "ocs_ci.ocs.exceptions.CommandFailed: Error during execution of command: oc -n openshift-storage patch cephblockpool pas-testing-rbd --type=merge -p '{"metadata":{"finalizers":null}}'. Error is Error from server (NotFound): cephblockpools.ceph.rook.io "pas-testing-rbd" not found"

ocs_ci.ocs.exceptions.CephHealthException: Can not create new filesystem

failed on teardown with "AttributeError: 'TestPvcMultiSnapshotPerformance' object has no attribute 'pvc_obj'"

ypersky1980 commented 6 months ago

PR with a fix https://github.com/red-hat-storage/ocs-ci/pull/9448

ypersky1980 commented 6 months ago

The PR was merged, therefore closing the issue.