search

krkn-chaos / krkn

Chaos and resiliency testing tool for Kubernetes with a focus on improving performance under failure conditions. A CNCF sandbox project.
Apache License 2.0
291 stars 102 forks source link

Ibm shut down #697

Closed paigerube14 closed 3 weeks ago

paigerube14 commented 2 months ago

Want to add in power-off/shut-down scenario for ibm cloud to have more coverage in prow 

2024-09-18 19:12:11,142 [INFO] Starting kraken
2024-09-18 19:12:11,163 [INFO] Initializing client to talk to the Kubernetes cluster
2024-09-18 19:12:11,163 [INFO] Generated a uuid for the run: b3c11551-57ef-4512-8d24-15b2c6c21218
2024-09-18 19:12:16,131 [INFO] Publishing kraken status at http://0.0.0.0:8081
2024-09-18 19:12:16,135 [INFO] Starting http server at http://0.0.0.0:8081

2024-09-18 19:12:16,135 [INFO] Fetching cluster info
2024-09-18 19:12:16,830 [INFO] 4.17.0-0.nightly-2024-09-17-194622
2024-09-18 19:12:16,832 [INFO] Server URL: https://api.prubenda-ibm4.ibmcloud.qe.devcluster.openshift.com:6443
2024-09-18 19:12:16,832 [INFO] Daemon mode not enabled, will run through 10 iterations

2024-09-18 19:12:16,832 [INFO] Executing scenarios for iteration 0
2024-09-18 19:12:16,833 [INFO] connection set up
2024-09-18 19:12:16,834 [INFO] response RUN
2024-09-18 19:12:16,841 [INFO] set ibm
2024-09-18 19:12:16,841 [INFO] iam authen
2024-09-18 19:12:16,843 [INFO] iam service1
2024-09-18 19:12:16,843 [INFO] iam service2
2024-09-18 19:12:16,844 [INFO] set cloud type
2024-09-18 19:12:17,274 [INFO] nodes ['prubenda-ibm4-twnhp-master-0', 'prubenda-ibm4-twnhp-master-1', 'prubenda-ibm4-twnhp-master-2', 'prubenda-ibm4-twnhp-worker-1-sfzfd', 'prubenda-ibm4-twnhp-worker-2-zj4fb', 'prubenda-ibm4-twnhp-worker-3-z5fc9']
2024-09-18 19:12:33,568 [INFO] get node id 0787_27d8902c-*****
2024-09-18 19:12:34,515 [INFO] get node id 0797_0ecead65-*****
2024-09-18 19:12:34,854 [INFO] get node id 07a7_37e05c1d-*****
2024-09-18 19:12:35,396 [INFO] get node id 0787_bf071ba4-*****
2024-09-18 19:12:35,720 [INFO] get node id 0797_da1e0b5f-*****
2024-09-18 19:12:36,749 [INFO] get node id 07a7_7756e7b5-*****
2024-09-18 19:12:36,750 [INFO] node id list ['0787_27d8902c-*****', '0797_0ecead65-*****', '07a7_37e05c1d-*****', '0787_bf071ba4-*****', '0797_da1e0b5f-*****', '07a7_7756e7b5-*****']
2024-09-18 19:12:36,750 [INFO] Starting cluster_shut_down scenario injection
2024-09-18 19:12:36,857 [INFO] nodes type <class 'str'>
2024-09-18 19:12:36,857 [INFO] pool type<class 'list'>
2024-09-18 19:12:37,691 [INFO] Stopped Instance -- '0787_27d8902c-*****'
2024-09-18 19:12:49,985 [INFO] Stopped Instance -- '07a7_7756e7b5-*****'
2024-09-18 19:13:22,518 [INFO] Stopped Instance -- '0797_da1e0b5f-*****'
2024-09-18 19:13:47,204 [INFO] Stopped Instance -- '0787_bf071ba4-*****'
2024-09-18 19:14:15,737 [INFO] Stopped Instance -- '07a7_37e05c1d-*****'
2024-09-18 19:14:37,105 [INFO] Stopped Instance -- '0797_0ecead65-*****'
2024-09-18 19:14:37,788 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:14:43,048 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:14:48,351 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:14:53,659 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:14:58,905 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:04,325 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:09,633 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:14,896 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:20,146 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:25,606 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:30,869 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:36,156 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:41,159 [INFO] Instance 0797_0ecead65-***** is still not stopped in allotted time
2024-09-18 19:15:42,431 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:47,689 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:52,952 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:15:58,185 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:03,454 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:08,743 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:14,011 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:19,575 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:24,850 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:30,094 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:35,548 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:40,849 [INFO] Instance 07a7_37e05c1d-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:45,854 [INFO] Instance 07a7_37e05c1d-***** is still not stopped in allotted time
2024-09-18 19:16:46,912 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:52,184 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:16:57,470 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:17:02,726 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:17:08,002 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:17:13,277 [INFO] Instance 0797_0ecead65-***** is still not stopped, sleeping for 5 seconds
2024-09-18 19:17:18,557 [INFO] Shutting down the cluster for the specified duration: 150
2024-09-18 19:19:48,564 [INFO] Restarting the nodes
2024-09-18 19:19:48,572 [INFO] nodes type <class 'str'>
2024-09-18 19:19:48,573 [INFO] pool type<class 'list'>
2024-09-18 19:19:49,277 [INFO] Started Instance -- '0797_0ecead65-*****'
2024-09-18 19:19:49,287 [INFO] Started Instance -- '0787_27d8902c-*****'
2024-09-18 19:19:49,307 [INFO] Started Instance -- '0797_da1e0b5f-*****'
2024-09-18 19:19:49,310 [INFO] Started Instance -- '0787_bf071ba4-*****'
2024-09-18 19:20:02,729 [INFO] Started Instance -- '07a7_37e05c1d-*****'
2024-09-18 19:20:34,659 [INFO] Started Instance -- '07a7_7756e7b5-*****'
2024-09-18 19:20:34,661 [INFO] Wait for each node to be running again
2024-09-18 19:20:38,550 [INFO] Instance 07a7_7756e7b5-***** is still not running, sleeping for 5 seconds
2024-09-18 19:20:43,986 [INFO] Instance 07a7_7756e7b5-***** is still not running, sleeping for 5 seconds
2024-09-18 19:20:49,062 [INFO] Waiting for 150s to allow cluster component initialization
2024-09-18 19:23:19,069 [INFO] Successfully injected cluster_shut_down scenario!
2024-09-18 19:23:19,071 [INFO] Waiting for the specified duration: 200
paigerube14 commented 1 month ago

@chaitanyaenr are you able to review this, I would like to add this to our tests as part of 4.18 coverage