feat(pod-delete): Updating failure condition for the pod-delete helper pod

[ispeakc0de]

Signed-off-by: shubhamchaudhary shubham.chaudhary@mayadata.io

What this PR does / why we need it:

• Updating the failure condition for the pod-delete helper pod

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Checklist

• [ ] Does this PR have a corresponding GitHub issue?
• [ ] Have you included relevant README for the chaoslib/experiment with details?
• [ ] Have you added debug messages where necessary?
• [ ] Have you added task comments where necessary?
• [ ] Have you tested the changes for possible failure conditions?
• [ ] Have you provided the positive & negative test logs for the litmusbook execution?
• [ ] Does the litmusbook ensure idempotency of cluster state?, i.e., is cluster restored to original state?
• [ ] Have you used non-shell/command modules for Kubernetes tasks?
• [ ] Have you (jinja) templatized custom scripts that is run by the litmusbook, if any?
• [ ] Have you (jinja) templatized Kubernetes deployment manifests used by the litmusbook, if any?
• [ ] Have you reused/created util functions instead of repeating tasks in the litmusbook?
• [ ] Do the artifacts follow the appropriate directory structure?
• [ ] Have you isolated storage (eg: OpenEBS) specific implementations, checks?
• [ ] Have you isolated platform (eg: baremetal kubeadm/openshift/aws/gcloud) specific implementations, checks?
• [ ] Are the ansible facts well defined? Is the scope explicitly set for playbook & included utils?
• [ ] Have you ensured minimum/careful usage of shell utilities (awk, grep, sed, cut, xargs etc.,)?
• [ ] Can the limtusbook be executed both from within & outside a container (configurable paths, no hardcode)?
• [ ] Can you suggest the minimal resource requirements for the litmusbook execution?
• [ ] Does the litmusbook job artifact carry comments/default options/range for the ENV tunables?
• [ ] Has the litmusbooks been linted?

Special notes for your reviewer:

[ispeakc0de]

2020-07-15 13:12:06.722836 ************ BRACE YOURSELF, EXPERIMENT BEGINS! ************ 

2020-07-15 13:12:10.443975 Step: Gathering Facts 
task path: /experiments/generic/pod_delete/pod_delete_ansible_logic.yml:2
META: ran handlers

2020-07-15 13:12:10.812951 Step: [PreReq] Identify the chaos util for pod-delete experiment 
task path: /experiments/generic/pod_delete/pod_delete_ansible_prerequisites.yml:1
details => {"changed": true, "checksum": "a4b0c755a1d92a8ea0226d8000cc0d63b310e6a1", "dest": "/tmp/chaosutil.yml", "gid": 0, "group": "root", "md5sum": "868bdb8c466c56152cc0a1e5d2b7e5dc", "mode": "0644", "owner": "root", "size": 62, "src": "/tmp/ansible-tmp-1594818730.49-271759930768610/source", "state": "file", "uid": 0}

2020-07-15 13:12:10.853956 Step: [PreReq]: Including the chaos util for the pod-delete experiment 
task path: /experiments/generic/pod_delete/pod_delete_ansible_logic.yml:28
details => {"ansible_facts": {"c_util": "/chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml"}, "ansible_included_var_files": ["/tmp/chaosutil.yml"], "changed": false}

2020-07-15 13:12:10.897447 Step: [PreReq]: Constructing the chaos result name 
task path: /experiments/generic/pod_delete/pod_delete_ansible_logic.yml:33
details => {"ansible_facts": {"c_result": "engine-pod-delete"}, "changed": false}
included: /utils/runtime/update_chaos_result_resource.yml for 127.0.0.1

2020-07-15 13:12:11.136077 Step: [PreReq]: Generate the chaos result CR to reflect SOT (Start of Test) 
task path: /utils/runtime/update_chaos_result_resource.yml:3
details => {"changed": true, "checksum": "a34d411ef886349103deda6687cd24e5d585f3d8", "dest": "/tmp/chaos-result.yml", "gid": 0, "group": "root", "md5sum": "677a8b05c749a8e697d7200361d03162", "mode": "0644", "owner": "root", "size": 305, "src": "/tmp/ansible-tmp-1594818730.98-144888948670927/source", "state": "file", "uid": 0}

2020-07-15 13:12:14.688685 Step: [PreReq]: Apply the chaos result CR for pod-delete experiment 
task path: /utils/runtime/update_chaos_result_resource.yml:13
details => {"changed": true, "cmd": "kubectl apply -f /tmp/chaos-result.yml -n shubham", "delta": "0:00:03.248053", "end": "2020-07-15 13:12:14.599678", "failed_when_result": false, "rc": 0, "start": "2020-07-15 13:12:11.351625", "stderr": "", "stderr_lines": [], "stdout": "chaosresult.litmuschaos.io/engine-pod-delete configured", "stdout_lines": ["chaosresult.litmuschaos.io/engine-pod-delete configured"]}

2020-07-15 13:12:14.814328 Step: [Info]: Display the application information passed via the test job 
task path: /experiments/generic/pod_delete/pod_delete_ansible_logic.yml:46
details => {
    "msg": [
        "The application info is as follows:", 
        "Namespace    : shubham", 
        "Label        : run=nginx", 
        "Ramp Time    : 5"
    ]
}
included: /utils/common/status_app_pod.yml for 127.0.0.1

2020-07-15 13:12:16.221414 Step: [Status]: Checking whether application pods are in running state 
task path: /utils/common/status_app_pod.yml:2
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pods -n shubham -l run=nginx -o custom-columns=:.status.phase --no-headers", "delta": "0:00:01.206264", "end": "2020-07-15 13:12:16.188695", "rc": 0, "start": "2020-07-15 13:12:14.982431", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}

2020-07-15 13:12:17.105165 Step: [Status]: Checking whether application containers are in running state 
task path: /utils/common/status_app_pod.yml:9
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pod -n shubham -l run=nginx --no-headers -o jsonpath='{.items[*].status.containerStatuses[*].ready}' | tr ' ' '\\n' | uniq", "delta": "0:00:00.676317", "end": "2020-07-15 13:12:17.087366", "rc": 0, "start": "2020-07-15 13:12:16.411049", "stderr": "", "stderr_lines": [], "stdout": "true", "stdout_lines": ["true"]}
included: /utils/common/generate-kubernetes-chaos-events.yml for 127.0.0.1

2020-07-15 13:12:17.555218 Step: [Event]: Obtain the current date in ISO8601 format 
task path: /utils/common/generate-kubernetes-chaos-events.yml:1
details => {"changed": true, "cmd": "date --iso-8601=seconds", "delta": "0:00:00.275290", "end": "2020-07-15 13:12:17.535976", "rc": 0, "start": "2020-07-15 13:12:17.260686", "stderr": "", "stderr_lines": [], "stdout": "2020-07-15T13:12:17+00:00", "stdout_lines": ["2020-07-15T13:12:17+00:00"]}

2020-07-15 13:12:17.614279 Step: set_fact 
task path: /utils/common/generate-kubernetes-chaos-events.yml:5
details => {"ansible_facts": {"ts": "2020-07-15T13:12:17Z"}, "changed": false}

2020-07-15 13:12:20.022080 Step: [Event]: Creating a PreChaosCheck event in the chaosengine 
task path: /utils/common/generate-kubernetes-chaos-events.yml:8
details => {"changed": true, "method": "patch", "result": {"apiVersion": "v1", "count": 1, "eventTime": null, "firstTimestamp": "2020-07-15T13:12:17Z", "involvedObject": {"apiVersion": "litmuschaos.io/v1alpha1", "kind": "ChaosEngine", "name": "engine", "namespace": "shubham", "uid": "0a91af9a-2fbf-4fc1-896e-838dbaadfcf9"}, "kind": "Event", "lastTimestamp": "2020-07-15T13:12:17Z", "message": "AUT is Running successfully", "metadata": {"creationTimestamp": "2020-07-15T12:59:57Z", "name": "event-\"PreChaosCheck\"", "namespace": "shubham", "resourceVersion": "516497", "selfLink": "/api/v1/namespaces/shubham/events/event-%22PreChaosCheck%22", "uid": "dd1982bb-fa20-4659-9d8e-6e14e76e6f49"}, "reason": "PreChaosCheck", "reportingComponent": "", "reportingInstance": "", "source": {"component": "pod-delete-kaxttb-rbnmc"}, "type": "Normal"}}
included: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml for 127.0.0.1

2020-07-15 13:12:20.173527 Step: [Prepare]: Deriving the chaos iterations 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:3
details => {"ansible_facts": {"c_iterations": "3"}, "changed": false}

2020-07-15 13:12:20.282553 Step: debug 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:20
details => {
    "msg": "***** Waiting for the ramp interval of 5s *****"
}

2020-07-15 13:12:25.601057 Step: [Ramp]: Waiting for the specified ramp time before injecting chaos 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:23
details => {"changed": false, "elapsed": 5, "path": null, "port": null, "search_regex": null, "state": "started"}

2020-07-15 13:12:26.099784 Step: [Prepare]: Generate a run id if not passed from the engine/experiment 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:29
details => {"changed": true, "cmd": "echo $(mktemp) | cut -d '.' -f 2 | cut -c -6", "delta": "0:00:00.319422", "end": "2020-07-15 13:12:26.085140", "rc": 0, "start": "2020-07-15 13:12:25.765718", "stderr": "", "stderr_lines": [], "stdout": "OR5lwv", "stdout_lines": ["OR5lwv"]}

2020-07-15 13:12:26.145490 Step: set_fact 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:33
details => {"ansible_facts": {"run_id": "or5lwv"}, "changed": false}

2020-07-15 13:12:26.923816 Step: Getting the serviceAccountName 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:37
details => {"changed": true, "cmd": "kubectl get pod pod-delete-kaxttb-rbnmc -n shubham -o custom-columns=:.spec.serviceAccountName --no-headers", "delta": "0:00:00.611101", "end": "2020-07-15 13:12:26.883487", "rc": 0, "start": "2020-07-15 13:12:26.272386", "stderr": "", "stderr_lines": [], "stdout": "litmus", "stdout_lines": ["litmus"]}

2020-07-15 13:12:27.175875 Step: [Prepare]: Including the litmus LIB for random pod deletion 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:45
details => {"changed": true, "checksum": "7e6e8324aa45c89919e7f1db2e0997710f0ee6a2", "dest": "/tmp/pod-delete-chaos.yml", "gid": 0, "group": "root", "md5sum": "637a353f4a55ad12a186b8997eb3ccd6", "mode": "0644", "owner": "root", "size": 1552, "src": "/tmp/ansible-tmp-1594818746.99-168664463716747/source", "state": "file", "uid": 0}

2020-07-15 13:12:30.197652 Step: [Prepare]: Create helper pod for pod delete chaos 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:53
details => {"changed": true, "cmd": "kubectl create -f /tmp/pod-delete-chaos.yml -n shubham", "delta": "0:00:02.729452", "end": "2020-07-15 13:12:30.021944", "rc": 0, "start": "2020-07-15 13:12:27.292492", "stderr": "", "stderr_lines": [], "stdout": "pod/pod-delete-or5lwv created", "stdout_lines": ["pod/pod-delete-or5lwv created"]}
FAILED - RETRYING: [Status]: Checking the running status of the helper pod (90 retries left).
FAILED - RETRYING: [Status]: Checking the running status of the helper pod (89 retries left).

2020-07-15 13:12:40.062927 Step: [Status]: Checking the running status of the helper pod 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:59
details => {"attempts": 3, "changed": true, "cmd": "kubectl get pods -l name=pod-delete-or5lwv -n shubham -o custom-columns=:.status.phase --no-headers", "delta": "0:00:01.666460", "end": "2020-07-15 13:12:39.988029", "rc": 0, "start": "2020-07-15 13:12:38.321569", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}
FAILED - RETRYING: [Status]: Checking whether helper pods is in completed state (40 retries left).
FAILED - RETRYING: [Status]: Checking whether helper pods is in completed state (39 retries left).
FAILED - RETRYING: [Status]: Checking whether helper pods is in completed state (38 retries left).
FAILED - RETRYING: [Status]: Checking whether helper pods is in completed state (37 retries left).
FAILED - RETRYING: [Status]: Checking whether helper pods is in completed state (36 retries left).

2020-07-15 13:13:36.220398 Step: [Status]: Checking whether helper pods is in completed state 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:67
details => {"attempts": 6, "changed": true, "cmd": "kubectl get pods -n shubham -l name=pod-delete-or5lwv -o custom-columns=:.status.phase --no-headers", "delta": "0:00:00.543964", "end": "2020-07-15 13:13:36.185133", "rc": 0, "start": "2020-07-15 13:13:35.641169", "stderr": "", "stderr_lines": [], "stdout": "Succeeded", "stdout_lines": ["Succeeded"]}

2020-07-15 13:13:37.385460 Step: [CleanUP]: Tear down pod delete helper pod 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:80
details => {"changed": true, "cmd": "kubectl delete -f /tmp/pod-delete-chaos.yml -n shubham", "delta": "0:00:00.818874", "end": "2020-07-15 13:13:37.343968", "rc": 0, "start": "2020-07-15 13:13:36.525094", "stderr": "", "stderr_lines": [], "stdout": "pod \"pod-delete-or5lwv\" deleted", "stdout_lines": ["pod \"pod-delete-or5lwv\" deleted"]}

2020-07-15 13:13:37.888895 Step: [Status]: Confirm that the pod delete helper pod is deleted 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:85
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pods -n shubham -l name=pod-delete-or5lwv -o custom-columns=:.status.phase --no-headers", "delta": "0:00:00.367256", "end": "2020-07-15 13:13:37.872476", "rc": 0, "start": "2020-07-15 13:13:37.505220", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

2020-07-15 13:13:37.936053 Step: debug 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:96
details => {
    "msg": "***** Waiting for the ramp interval of 5s *****"
}

2020-07-15 13:13:43.128388 Step: [Ramp]: Waiting for the specified ramp time after injecting chaos 
task path: /chaoslib/litmus/pod_delete/pod_failure_by_litmus.yml:99
details => {"changed": false, "elapsed": 5, "path": null, "port": null, "search_regex": null, "state": "started"}
included: /utils/common/status_app_pod.yml for 127.0.0.1

2020-07-15 13:13:44.210466 Step: [Status]: Checking whether application pods are in running state 
task path: /utils/common/status_app_pod.yml:2
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pods -n shubham -l run=nginx -o custom-columns=:.status.phase --no-headers", "delta": "0:00:00.668039", "end": "2020-07-15 13:13:44.184727", "rc": 0, "start": "2020-07-15 13:13:43.516688", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}

2020-07-15 13:13:44.768379 Step: [Status]: Checking whether application containers are in running state 
task path: /utils/common/status_app_pod.yml:9
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pod -n shubham -l run=nginx --no-headers -o jsonpath='{.items[*].status.containerStatuses[*].ready}' | tr ' ' '\\n' | uniq", "delta": "0:00:00.394470", "end": "2020-07-15 13:13:44.751574", "rc": 0, "start": "2020-07-15 13:13:44.357104", "stderr": "", "stderr_lines": [], "stdout": "true", "stdout_lines": ["true"]}
included: /utils/common/generate-kubernetes-chaos-events.yml for 127.0.0.1

2020-07-15 13:13:45.243592 Step: [Event]: Obtain the current date in ISO8601 format 
task path: /utils/common/generate-kubernetes-chaos-events.yml:1
details => {"changed": true, "cmd": "date --iso-8601=seconds", "delta": "0:00:00.302612", "end": "2020-07-15 13:13:45.227469", "rc": 0, "start": "2020-07-15 13:13:44.924857", "stderr": "", "stderr_lines": [], "stdout": "2020-07-15T13:13:45+00:00", "stdout_lines": ["2020-07-15T13:13:45+00:00"]}

2020-07-15 13:13:45.291349 Step: set_fact 
task path: /utils/common/generate-kubernetes-chaos-events.yml:5
details => {"ansible_facts": {"ts": "2020-07-15T13:13:45Z"}, "changed": false}

2020-07-15 13:13:49.144532 Step: [Event]: Creating a PostChaosCheck event in the chaosengine 
task path: /utils/common/generate-kubernetes-chaos-events.yml:8
details => {"changed": true, "method": "create", "result": {"apiVersion": "v1", "count": 1, "eventTime": null, "firstTimestamp": "2020-07-15T13:13:45Z", "involvedObject": {"apiVersion": "litmuschaos.io/v1alpha1", "kind": "ChaosEngine", "name": "engine", "namespace": "shubham", "uid": "0a91af9a-2fbf-4fc1-896e-838dbaadfcf9"}, "kind": "Event", "lastTimestamp": "2020-07-15T13:13:45Z", "message": "AUT is Running successfully", "metadata": {"creationTimestamp": "2020-07-15T13:13:48Z", "name": "event-\"PostChaosCheck\"", "namespace": "shubham", "resourceVersion": "516642", "selfLink": "/api/v1/namespaces/shubham/events/event-%22PostChaosCheck%22", "uid": "7f4d78fa-7c5d-4838-ae30-1f7174104b7e"}, "reason": "PostChaosCheck", "reportingComponent": "", "reportingInstance": "", "source": {"component": "pod-delete-kaxttb-rbnmc"}, "type": "Normal"}}

2020-07-15 13:13:49.199776 Step: set_fact 
task path: /experiments/generic/pod_delete/pod_delete_ansible_logic.yml:98
details => {"ansible_facts": {"flag": "Pass"}, "changed": false}

2020-07-15 13:13:49.251060 Step: [Result]: Getting the final result of pod-delete experiment 
task path: /experiments/generic/pod_delete/pod_delete_ansible_logic.yml:101
details => {
    "msg": "pod-delete experiment has been Passed"
}
included: /utils/runtime/getting_failure_step.yml for 127.0.0.1

2020-07-15 13:13:49.536438 Step: [Failure-Detection]: Recording the name of failed step 
task path: /utils/runtime/getting_failure_step.yml:44
details => {"ansible_facts": {"failStep": "N/A"}, "changed": false}
included: /utils/common/generate-kubernetes-chaos-events.yml for 127.0.0.1

2020-07-15 13:13:50.074693 Step: [Event]: Obtain the current date in ISO8601 format 
task path: /utils/common/generate-kubernetes-chaos-events.yml:1
details => {"changed": true, "cmd": "date --iso-8601=seconds", "delta": "0:00:00.292392", "end": "2020-07-15 13:13:50.057497", "rc": 0, "start": "2020-07-15 13:13:49.765105", "stderr": "", "stderr_lines": [], "stdout": "2020-07-15T13:13:50+00:00", "stdout_lines": ["2020-07-15T13:13:50+00:00"]}

2020-07-15 13:13:50.138340 Step: set_fact 
task path: /utils/common/generate-kubernetes-chaos-events.yml:5
details => {"ansible_facts": {"ts": "2020-07-15T13:13:50Z"}, "changed": false}

2020-07-15 13:14:10.564536 Step: [Event]: Creating a Summary event in the chaosengine 
task path: /utils/common/generate-kubernetes-chaos-events.yml:8
details => {"changed": true, "method": "patch", "result": {"apiVersion": "v1", "count": 1, "eventTime": null, "firstTimestamp": "2020-07-15T13:13:50Z", "involvedObject": {"apiVersion": "litmuschaos.io/v1alpha1", "kind": "ChaosEngine", "name": "engine", "namespace": "shubham", "uid": "0a91af9a-2fbf-4fc1-896e-838dbaadfcf9"}, "kind": "Event", "lastTimestamp": "2020-07-15T13:13:50Z", "message": "pod-delete Experiment Passed!", "metadata": {"creationTimestamp": "2020-07-15T13:02:46Z", "name": "event-\"Summary\"", "namespace": "shubham", "resourceVersion": "516657", "selfLink": "/api/v1/namespaces/shubham/events/event-%22Summary%22", "uid": "07056af8-afc5-4e11-950e-405ce7808b16"}, "reason": "Summary", "reportingComponent": "", "reportingInstance": "", "source": {"component": "pod-delete-kaxttb-rbnmc"}, "type": "Normal"}}
included: /utils/runtime/update_chaos_result_resource.yml for 127.0.0.1

2020-07-15 13:14:11.848925 Step: [Result]: Update the chaos result CR to reflect EOT (End of Test) 
task path: /utils/runtime/update_chaos_result_resource.yml:23
details => {"changed": true, "checksum": "cb7f99782bad762ea2e34596093ddf11d189beef", "dest": "/tmp/chaos-result.yml", "gid": 0, "group": "root", "md5sum": "38bf5c6de3e29f5d08de11feed9d09c0", "mode": "0644", "owner": "root", "size": 324, "src": "/tmp/ansible-tmp-1594818850.84-158636619554727/source", "state": "file", "uid": 0}

2020-07-15 13:14:12.937300 Step: [The END]: Apply the chaos result CR for pod-delete experiment 
task path: /utils/runtime/update_chaos_result_resource.yml:33
details => {"changed": true, "cmd": "kubectl apply -f /tmp/chaos-result.yml -n shubham", "delta": "0:00:00.864601", "end": "2020-07-15 13:14:12.917031", "failed_when_result": false, "rc": 0, "start": "2020-07-15 13:14:12.052430", "stderr": "", "stderr_lines": [], "stdout": "chaosresult.litmuschaos.io/engine-pod-delete configured", "stdout_lines": ["chaosresult.litmuschaos.io/engine-pod-delete configured"]}
META: ran handlers
META: ran handlers

2020-07-15 13:14:12.947588 ************ RELAX, EXPERIMENT ENDS! ************ 
127.0.0.1                  : ok=48   changed=23   unreachable=0    failed=0