test_deletion_sync_after_node_restart[noobaa-core] is failing on IBM Power

[Pooja-Soni78]

Below test case is failing on ODF 4.15 on IBM Power - tests/manage/mcg/test_log_based_bucket_replication.py::TestLogBasedBucketReplication::test_deletion_sync_after_node_restart[noobaa-core]

Error being -

>               raise CommandFailed(
                    f"Error during execution of command: {masked_cmd}."
                    f"\nError is {masked_stderr}"
                )
E               ocs_ci.ocs.exceptions.CommandFailed: Error during execution of command: oc -n openshift-storage rsh s3cli-0 sh -c "AWS_CA_BUNDLE=/cert/service-ca.crt AWS_ACCESS_KEY_ID=***** AWS_SECRET_ACCESS_KEY=***** AWS_DEFAULT_REGION=us-east-2 aws s3 --endpoint=***** ls s3://oc-bucket-02fe80d2631049718b7a694d0c4b6f".
E               Error is error: Internal error occurred: error executing command in container: container is not created or running

ocs_ci/utility/utils.py:664: CommandFailed

[Pooja-Soni78]

logs - test_deletion_sync_after_node_restart[noobaa-core].log

[sagihirshfeld]

We can't determine for certain without the must-gather logs, but it looks like the AWS CLI pod (s3cli-0) resided on the same node as noobaa-core during this run and was still starting up post node-reset while the noobaa-core was already up, which caused the test to crash when attempting to use it via oc rsh.

Adding a validation to make sure the AWS CLI pod is ready before proceeding should prevent this error - I'll set the PR with the fix up shortly.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs.

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.