search

red-hat-storage / ocs-ci

https://ocs-ci.readthedocs.io/en/latest/
MIT License
108 stars 166 forks source link

Deletion of gcp-backingstore-f5d37497b710466cb98be43 failed due to its state; Retrying for 4 hours #9972

Closed petr-balogh closed 2 weeks ago

petr-balogh commented 4 months ago 
2024-06-21 13:12:27  07:12:27 - MainThread - ocs_ci.utility.utils - ERROR  - Exception raised during iteration: 
2024-06-21 13:12:27  Traceback (most recent call last):
2024-06-21 13:12:27    File "/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/ocs/resources/backingstore.py", line 90, in _oc_deletion_flow
2024-06-21 13:12:27      OCP(
2024-06-21 13:12:27    File "/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/ocs/ocp.py", line 408, in delete
2024-06-21 13:12:27      return self.exec_oc_cmd(command, timeout=timeout)
2024-06-21 13:12:27    File "/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/ocs/ocp.py", line 178, in exec_oc_cmd
2024-06-21 13:12:27      out = run_cmd(
2024-06-21 13:12:27    File "/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/utils.py", line 486, in run_cmd
2024-06-21 13:12:27      completed_process = exec_cmd(
2024-06-21 13:12:27    File "/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/utils.py", line 666, in exec_cmd
2024-06-21 13:12:27      raise CommandFailed(
2024-06-21 13:12:27  ocs_ci.ocs.exceptions.CommandFailed: Error during execution of command: oc -n openshift-storage delete Backingstore gcp-backingstore-f5d37497b710466cb98be43.
2024-06-21 13:12:27  Error is Error from server: admission webhook "admissionwebhook.noobaa.io" denied the request: cannot complete because objects in Backingstore "gcp-backingstore-f5d37497b710466cb98be43" are still being deleted, Please try later
2024-06-21 13:12:27  
2024-06-21 13:12:27  
2024-06-21 13:12:27  During handling of the above exception, another exception occurred:
2024-06-21 13:12:27  
2024-06-21 13:12:27  Traceback (most recent call last):
2024-06-21 13:12:27    File "/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/utility/utils.py", line 1292, in __iter__
2024-06-21 13:12:27      yield self.func(*self.func_args, **self.func_kwargs)
2024-06-21 13:12:27    File "/home/jenkins/workspace/qe-deploy-ocs-cluster-prod/ocs-ci/ocs_ci/ocs/resources/backingstore.py", line 115, in _oc_deletion_flow
2024-06-21 13:12:27      raise ObjectsStillBeingDeletedException
2024-06-21 13:12:27  ocs_ci.ocs.exceptions.ObjectsStillBeingDeletedException
2024-06-21 13:12:27  07:12:27 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 20 seconds before next iteration
2024-06-21 13:12:32  07:12:31 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:12:47  07:12:47 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage delete Backingstore gcp-backingstore-f5d37497b710466cb98be43
2024-06-21 13:12:47  07:12:47 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server: admission webhook "admissionwebhook.noobaa.io" denied the request: cannot complete because pool "gcp-backingstore-f5d37497b710466cb98be43" in "DEFAULT_RESOURCE" state
2024-06-21 13:12:47  
2024-06-21 13:12:47  07:12:47 - MainThread - ocs_ci.ocs.resources.backingstore - WARNING  - Deletion of gcp-backingstore-f5d37497b710466cb98be43 failed due to its state; Retrying
2024-06-21 13:12:47  07:12:47 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 20 seconds before next iteration
2024-06-21 13:12:57  07:12:56 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:03  07:13:02 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:07  07:13:07 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:08  07:13:07 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage delete Backingstore gcp-backingstore-f5d37497b710466cb98be43
2024-06-21 13:13:08  07:13:08 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server: admission webhook "admissionwebhook.noobaa.io" denied the request: cannot complete because pool "gcp-backingstore-f5d37497b710466cb98be43" in "DEFAULT_RESOURCE" state
2024-06-21 13:13:08  
2024-06-21 13:13:08  07:13:08 - MainThread - ocs_ci.ocs.resources.backingstore - WARNING  - Deletion of gcp-backingstore-f5d37497b710466cb98be43 failed due to its state; Retrying
2024-06-21 13:13:08  07:13:08 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 20 seconds before next iteration
2024-06-21 13:13:12  07:13:12 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:17  07:13:17 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:23  07:13:22 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:28  07:13:27 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:28  07:13:28 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage delete Backingstore gcp-backingstore-f5d37497b710466cb98be43
2024-06-21 13:13:28  07:13:28 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server: admission webhook "admissionwebhook.noobaa.io" denied the request: cannot complete because pool "gcp-backingstore-f5d37497b710466cb98be43" in "DEFAULT_RESOURCE" state
2024-06-21 13:13:28  
2024-06-21 13:13:28  07:13:28 - MainThread - ocs_ci.ocs.resources.backingstore - WARNING  - Deletion of gcp-backingstore-f5d37497b710466cb98be43 failed due to its state; Retrying
2024-06-21 13:13:28  07:13:28 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 20 seconds before next iteration
2024-06-21 13:13:33  07:13:32 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:38  07:13:37 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:42  07:13:42 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:47  07:13:47 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 13:13:49  07:13:48 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage delete Backingstore gcp-backingstore-f5d37497b710466cb98be43
2024-06-21 13:13:49  07:13:49 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server: admission webhook "admissionwebhook.noobaa.io" denied the request: cannot complete because pool "gcp-backingstore-f5d37497b710466cb98be43" in "DEFAULT_RESOURCE" state
2024-06-21 13:13:49  
....
...
2024-06-21 16:32:16  10:32:16 - MainThread - ocs_ci.ocs.resources.backingstore - WARNING  - Deletion of gcp-backingstore-f5d37497b710466cb98be43 failed due to its state; Retrying
2024-06-21 16:32:16  10:32:16 - MainThread - ocs_ci.utility.utils - INFO  - Going to sleep for 20 seconds before next iteration
2024-06-21 16:32:21  10:32:21 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 16:32:26  10:32:26 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 16:32:32  10:32:31 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 16:32:37  10:32:36 - MainThread - ocs_ci.utility.utils - INFO  - Executing command: oc -n openshift-storage delete Backingstore gcp-backingstore-f5d37497b710466cb98be43
2024-06-21 16:32:37  10:32:36 - Thread-2 - ocs_ci.utility.retry - WARNING  - list index out of range, Retrying in 5 seconds...
2024-06-21 16:32:37  10:32:36 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server: admission webhook "admissionwebhook.noobaa.io" denied the request: cannot complete because pool "gcp-backingstore-f5d37497b710466cb98be43" in "DEFAULT_RESOURCE" state
2024-06-21 16:32:37

Job: https://url.corp.redhat.com/e8fc9e9

Poduction job: job/qe-trigger-vsphere-upi-1az-rhcos-external-3m-3w-upgrade-ocs-auto/113/

Logs: https://url.corp.redhat.com/465e6aa

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs.

github-actions[bot] commented 2 weeks ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.