test_unidirectional_bucketclass_replication failed with ubprocess.TimeoutExpired: Command '['/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/data/mcg-cli', 'obc', 'create', '--exact', 'cli-bucket-2127ef2200a247dc8f3b0ae371020', '--bucketclass', 'cli-bucketclass-ff19c624016b4c16b29a1f22', '-n', 'openshift-storage']' timed out after 600 second

[ebenahar]

Run details:

URL: https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#OCS/launches/362/6433/263940/263971/263975/log Run ID: 1668626119 Test Case: test_unidirectional_bucketclass_replication ODF Build: 4.12.0-113 OCP Version: 4.12 Job name: VSPHERE6 UPI 1AZ RHCOS MCG Only Deployment 3M 3W tier1 or tier_after_upgrade post upgrade Jenkins job: https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/6085/ Logs URL: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-008vu1rmcg33-ua/j-008vu1rmcg33-ua_20221116T143051/logs/

Failure Details:

Message: subprocess.TimeoutExpired: Command '['/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/data/mcg-cli', 'obc', 'create', '--exact', 'cli-bucket-2127ef2200a247dc8f3b0ae371020', '--bucketclass', 'cli-bucketclass-ff19c624016b4c16b29a1f22', '-n', 'openshift-storage']' timed out after 600 seconds
Type: None

Text:
self = <tests.manage.mcg.test_bucket_replication.TestReplication object at 0x7fe846e8ad90>
awscli_pod_session = <ocs_ci.ocs.resources.pod.Pod object at 0x7fe866810c40>
mcg_obj_session = <ocs_ci.ocs.resources.mcg.MCG object at 0x7fe84ea61130>
bucket_factory = <function bucket_factory_fixture.<locals>._create_buckets at 0x7fe846fd4dc0>
source_bucketclass = {'backingstore_dict': {'aws': [(1, 'eu-central-1')]}, 'interface': 'CLI', 'replication_policy': ('basic-replication-rule', 'oc-bucket-902e2dd4f216497caa1d26e82886ac', None)}
target_bucketclass = {'backingstore_dict': {'gcp': [(1, None)]}, 'interface': 'OC'}

    @pytest.mark.parametrize(
        argnames=["source_bucketclass", "target_bucketclass"],
        argvalues=[
            pytest.param(
                {
                    "interface": "OC",
                    "backingstore_dict": {"aws": [(1, "eu-central-1")]},
                },
                {"interface": "OC", "backingstore_dict": {"azure": [(1, None)]}},
                marks=[tier2, pytest.mark.polarion_id("OCS-2680")],
            ),
            pytest.param(
                {
                    "interface": "CLI",
                    "backingstore_dict": {"azure": [(1, None)]},
                },
                {
                    "interface": "CLI",
                    "backingstore_dict": {"aws": [(1, "eu-central-1")]},
                },
                marks=[tier2, pytest.mark.polarion_id("OCS-2684")],
            ),
            pytest.param(
                {
                    "interface": "CLI",
                    "backingstore_dict": {"aws": [(1, "eu-central-1")]},
                },
                {
                    "interface": "OC",
                    "backingstore_dict": {"gcp": [(1, None)]},
                },
                marks=[tier1],
            ),
        ],
        ids=["AWStoAZURE-BC-OC", "AZUREtoAWS-BC-CLI", "AWStoGCP-BC-Hybrid"],
    )
    def test_unidirectional_bucketclass_replication(
        self,
        awscli_pod_session,
        mcg_obj_session,
        bucket_factory,
        source_bucketclass,
        target_bucketclass,
    ):
        """
        Test unidirectional bucketclass replication using CLI and YAML

        """
        target_bucket_name = bucket_factory(bucketclass=target_bucketclass)[0].name
        source_bucketclass["replication_policy"] = (
            "basic-replication-rule",
            target_bucket_name,
            None,
        )
>       source_bucket_name = bucket_factory(1, bucketclass=source_bucketclass)[0].name

tests/manage/mcg/test_bucket_replication.py:351: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/conftest.py:2560: in _create_buckets
    created_bucket = BUCKET_MAP[interface.lower()](
ocs_ci/ocs/resources/objectbucket.py:340: in __init__
    self.mcg.exec_mcg_cmd(
ocs_ci/ocs/resources/mcg.py:871: in exec_mcg_cmd
    result = exec_cmd(
ocs_ci/utility/utils.py:606: in exec_cmd
    completed_process = subprocess.run(
/usr/lib64/python3.8/subprocess.py:495: in run
    stdout, stderr = process.communicate(input, timeout=timeout)
/usr/lib64/python3.8/subprocess.py:1028: in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
/usr/lib64/python3.8/subprocess.py:1869: in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <subprocess.Popen object at 0x7fe846fe12b0>, endtime = 24253.070425489
orig_timeout = 600, stdout_seq = []
stderr_seq = [b'time="2022-11-16T21:04:36Z" level=info msg="\xe2\x9c\x85 Exists: StorageClass \\"openshift-storage.noobaa.io\\"\\n"...4:39Z" level=info msg="\xe2\x8f\xb3 OBC \\"cli-bucket-2127ef2200a247dc8f3b0ae371020\\" Phase is \\"Pending\\""\n', ...]
skip_check_and_raise = False

    def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq,
                       skip_check_and_raise=False):
        """Convenience for checking if a timeout has expired."""
        if endtime is None:
            return
        if skip_check_and_raise or _time() > endtime:
>           raise TimeoutExpired(
                    self.args, orig_timeout,
                    output=b''.join(stdout_seq) if stdout_seq else None,
                    stderr=b''.join(stderr_seq) if stderr_seq else None)
E           subprocess.TimeoutExpired: Command '['/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/data/mcg-cli', 'obc', 'create', '--exact', 'cli-bucket-2127ef2200a247dc8f3b0ae371020', '--bucketclass', 'cli-bucketclass-ff19c624016b4c16b29a1f22', '-n', 'openshift-storage']' timed out after 600 seconds

/usr/lib64/python3.8/subprocess.py:1072: TimeoutExpired

[ebenahar]

This issue was encountered once again. Run details:

URL: https://reportportal-ocs4.apps.ocp-c1.prod.psi.redhat.com/ui/#OCS/launches/362/6301/257429/257477/257481/log Run ID: 1668070972 Test Case: test_unidirectional_bucketclass_replication ODF Build: 4.12.0-91 OCP Version: 4.12 Job name: VSPHERE6 IPI 1AZ RHCOS VSAN 3M 3W tier2 Jenkins job: https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster-prod/6027/ Logs URL: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-031vi1cs33-t2/j-031vi1cs33-t2_20221110T073314/logs/ Error Message: ubprocess.TimeoutExpired: Command '['/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/data/mcg-cli', 'obc', 'create', '--exact', 'cli-bucket-974f109e5e04440ca343a60670f54', '--bucketclass', 'cli-bucketclass-f820e08295ed4794b5adc005', '-n', 'openshift-storage']' timed out after 600 second

[sagihirshfeld]

Both were categorized as System Issues after the logs revealed node failures