feat(kubelet-service-kill): Adding ability to pass nodeName explicitly in kubelet-service-kill

[ispeakc0de]

Signed-off-by: shubhamchaudhary shubham.chaudhary@mayadata.io

What this PR does / why we need it:

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Checklist

• [ ] Does this PR have a corresponding GitHub issue?
• [ ] Have you included relevant README for the chaoslib/experiment with details?
• [ ] Have you added debug messages where necessary?
• [ ] Have you added task comments where necessary?
• [ ] Have you tested the changes for possible failure conditions?
• [ ] Have you provided the positive & negative test logs for the litmusbook execution?
• [ ] Does the litmusbook ensure idempotency of cluster state?, i.e., is cluster restored to original state?
• [ ] Have you used non-shell/command modules for Kubernetes tasks?
• [ ] Have you (jinja) templatized custom scripts that is run by the litmusbook, if any?
• [ ] Have you (jinja) templatized Kubernetes deployment manifests used by the litmusbook, if any?
• [ ] Have you reused/created util functions instead of repeating tasks in the litmusbook?
• [ ] Do the artifacts follow the appropriate directory structure?
• [ ] Have you isolated storage (eg: OpenEBS) specific implementations, checks?
• [ ] Have you isolated platform (eg: baremetal kubeadm/openshift/aws/gcloud) specific implementations, checks?
• [ ] Are the ansible facts well defined? Is the scope explicitly set for playbook & included utils?
• [ ] Have you ensured minimum/careful usage of shell utilities (awk, grep, sed, cut, xargs etc.,)?
• [ ] Can the limtusbook be executed both from within & outside a container (configurable paths, no hardcode)?
• [ ] Can you suggest the minimal resource requirements for the litmusbook execution?
• [ ] Does the litmusbook job artifact carry comments/default options/range for the ENV tunables?
• [ ] Has the litmusbooks been linted?

Special notes for your reviewer:

[ispeakc0de]

2020-07-15 05:05:18.016391 ************ BRACE YOURSELF, EXPERIMENT BEGINS! ************ 

2020-07-15 05:05:33.614289 Step: Gathering Facts 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:2
META: ran handlers

2020-07-15 05:05:38.671121 Step: [PreReq] Identify the chaos util for kubelet-service-kill experiment 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_prerequisites.yml:1
details => {"changed": true, "checksum": "2d76d919fa3c20d76eff0b8e3148897f4319c491", "dest": "/tmp/chaosutil.yml", "gid": 0, "group": "root", "md5sum": "c855258e0f920687b5ce930bafa965a7", "mode": "0644", "owner": "root", "size": 73, "src": "/tmp/ansible-tmp-1594789536.95-224355542286051/source", "state": "file", "uid": 0}

2020-07-15 05:05:38.720220 Step: [PreReq]: Including the chaos util for the kubelet-service-kill experiment 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:28
details => {"ansible_facts": {"c_util": "/chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml"}, "ansible_included_var_files": ["/tmp/chaosutil.yml"], "changed": false}
included: /utils/runtime/update_chaos_result_resource.yml for 127.0.0.1

2020-07-15 05:05:39.014171 Step: [PreReq]: Generate the chaos result CR to reflect SOT (Start of Test) 
task path: /utils/runtime/update_chaos_result_resource.yml:3
details => {"changed": true, "checksum": "c8d58ff872d1de2e341f3c976653b9fca6563b3b", "dest": "/tmp/chaos-result.yml", "gid": 0, "group": "root", "md5sum": "ef1be1ec3abaf3e192d4ca2d0855b73b", "mode": "0644", "owner": "root", "size": 276, "src": "/tmp/ansible-tmp-1594789538.83-269181674567824/source", "state": "file", "uid": 0}

2020-07-15 05:07:45.875821 Step: [PreReq]: Apply the chaos result CR for kubelet-service-kill experiment 
task path: /utils/runtime/update_chaos_result_resource.yml:13
details => {"changed": true, "cmd": "kubectl apply -f /tmp/chaos-result.yml -n shubham", "delta": "0:02:05.527506", "end": "2020-07-15 05:07:44.766916", "failed_when_result": false, "rc": 0, "start": "2020-07-15 05:05:39.239410", "stderr": "Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply", "stderr_lines": ["Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply"], "stdout": "chaosresult.litmuschaos.io/kubelet-service-kill configured", "stdout_lines": ["chaosresult.litmuschaos.io/kubelet-service-kill configured"]}

2020-07-15 05:07:46.475810 Step: [Info]: Display the application information passed via the test job 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:46
details => {
    "msg": [
        "The application info is as follows:", 
        "Namespace    : shubham", 
        "Label        : run=nginx", 
        "Ramp Time    : ", 
        "Node name    : minikube"
    ]
}
included: /utils/common/status_app_pod.yml for 127.0.0.1

2020-07-15 05:07:50.807026 Step: [Status]: Checking whether application pods are in running state 
task path: /utils/common/status_app_pod.yml:2
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pods -n shubham -l run=nginx -o custom-columns=:.status.phase --no-headers", "delta": "0:00:03.688615", "end": "2020-07-15 05:07:50.653962", "rc": 0, "start": "2020-07-15 05:07:46.965347", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}

2020-07-15 05:07:54.308059 Step: [Status]: Checking whether application containers are in running state 
task path: /utils/common/status_app_pod.yml:9
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pod -n shubham -l run=nginx --no-headers -o jsonpath='{.items[*].status.containerStatuses[*].ready}' | tr ' ' '\\n' | uniq", "delta": "0:00:02.929706", "end": "2020-07-15 05:07:54.027913", "rc": 0, "start": "2020-07-15 05:07:51.098207", "stderr": "", "stderr_lines": [], "stdout": "true", "stdout_lines": ["true"]}
included: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml for 127.0.0.1

2020-07-15 05:07:55.014502 Step: [Prepare]: Record the application node name 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:33
details => {"ansible_facts": {"app_node": "minikube"}, "changed": false}

2020-07-15 05:07:56.079396 Step: [Prepare:] Generate a run id if not passed from the engine/experiment 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:40
details => {"changed": true, "cmd": "echo $(mktemp) | cut -d '.' -f 2 | cut -c -6", "delta": "0:00:00.322472", "end": "2020-07-15 05:07:56.062832", "rc": 0, "start": "2020-07-15 05:07:55.740360", "stderr": "", "stderr_lines": [], "stdout": "OtRFLc", "stdout_lines": ["OtRFLc"]}

2020-07-15 05:07:56.130941 Step: set_fact 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:44
details => {"ansible_facts": {"run_id": "otrflc"}, "changed": false}

2020-07-15 05:07:56.639950 Step: [Prepare]: Patch the run_id to kubelet service kill helper pod template 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:57
details => {"changed": true, "checksum": "b65e38e3d225a7b261f80cb58372dce026e98557", "dest": "/tmp/kubelet-service-kill.yml", "gid": 0, "group": "root", "md5sum": "9e95b0ed76011e88f708d956a4e88c17", "mode": "0644", "owner": "root", "size": 789, "src": "/tmp/ansible-tmp-1594789676.22-5820095282906/source", "state": "file", "uid": 0}

2020-07-15 05:07:56.682340 Step: set_fact 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:65
details => {"ansible_facts": {"pod_running_status": ""}, "changed": false}
included: /utils/common/create_chaos_pod.yml for 127.0.0.1 => (item=1)
included: /utils/common/create_chaos_pod.yml for 127.0.0.1 => (item=2)
included: /utils/common/create_chaos_pod.yml for 127.0.0.1 => (item=3)

2020-07-15 05:08:01.241306 Step: [Inject]: Creating the chaos pod in shubham namespace 
task path: /utils/common/create_chaos_pod.yml:3
details => {"changed": true, "cmd": "kubectl apply -f /tmp/kubelet-service-kill.yml -n shubham", "delta": "0:00:04.223659", "end": "2020-07-15 05:08:01.105298", "rc": 0, "start": "2020-07-15 05:07:56.881639", "stderr": "", "stderr_lines": [], "stdout": "pod/service-kill-otrflc created", "stdout_lines": ["pod/service-kill-otrflc created"]}
FAILED - RETRYING: [Status]: Checking the running status of the chaos pod (90 retries left).

2020-07-15 05:08:06.709343 Step: [Status]: Checking the running status of the chaos pod 
task path: /utils/common/create_chaos_pod.yml:10
details => {"attempts": 2, "changed": true, "cmd": "kubectl get pods -l name=service-kill-otrflc -n shubham -o custom-columns=:.status.phase --no-headers", "delta": "0:00:01.945772", "end": "2020-07-15 05:08:06.690519", "rc": 0, "start": "2020-07-15 05:08:04.744747", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}

2020-07-15 05:08:06.819326 Step: [Record]: Mark the creation of chaos pod successfully 
task path: /utils/common/create_chaos_pod.yml:43
details => {"ansible_facts": {"pod_running_status": "Running"}, "changed": false}
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (90 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (89 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (88 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (87 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (86 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (85 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (84 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (83 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (82 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (81 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (80 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (79 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (78 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (77 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (76 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (75 retries left).
FAILED - RETRYING: [Status]: Waiting for node to get in NotReady state (74 retries left).

2020-07-15 05:09:03.740029 Step: [Status]: Waiting for node to get in NotReady state 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:83
details => {"attempts": 18, "changed": true, "cmd": "kubectl get nodes minikube --no-headers | awk '{print$2}'", "delta": "0:00:01.235778", "end": "2020-07-15 05:09:03.678218", "rc": 0, "start": "2020-07-15 05:09:02.442440", "stderr": "", "stderr_lines": [], "stdout": "NotReady", "stdout_lines": ["NotReady"]}

2020-07-15 05:10:40.714453 Step: [Wait]: Wait for the chaos duration of 90s 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:93
details => {"changed": false, "elapsed": 90, "path": null, "port": null, "search_regex": null, "state": "started"}

2020-07-15 05:11:33.273101 Step: [Status]: Checking the node status after chaos 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:97
details => {"attempts": 1, "changed": true, "cmd": "kubectl get nodes minikube --no-headers | awk '{print$2}'", "delta": "0:00:29.171775", "end": "2020-07-15 05:11:25.097928", "rc": 0, "start": "2020-07-15 05:10:55.926153", "stderr": "", "stderr_lines": [], "stdout": "Ready", "stdout_lines": ["Ready"]}

2020-07-15 05:12:10.684766 Step: [CleanUP]: Tear down service kill infra 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:107
details => {"changed": true, "cmd": "kubectl delete -f /tmp/kubelet-service-kill.yml -n shubham", "delta": "0:00:22.255725", "end": "2020-07-15 05:12:10.555866", "rc": 0, "start": "2020-07-15 05:11:48.300141", "stderr": "", "stderr_lines": [], "stdout": "pod \"service-kill-otrflc\" deleted", "stdout_lines": ["pod \"service-kill-otrflc\" deleted"]}

2020-07-15 05:12:11.358562 Step: [Status]: Confirm that the svc chaos helper pod is teminated successfully 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:114
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pod -l name=service-kill-otrflc --no-headers -o custom-columns=:status.phase -n shubham | sort | uniq", "delta": "0:00:00.417899", "end": "2020-07-15 05:12:11.333119", "rc": 0, "start": "2020-07-15 05:12:10.915220", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
included: /utils/common/status_app_pod.yml for 127.0.0.1

2020-07-15 05:12:11.950919 Step: [Status]: Checking whether application pods are in running state 
task path: /utils/common/status_app_pod.yml:2
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pods -n shubham -l run=nginx -o custom-columns=:.status.phase --no-headers", "delta": "0:00:00.401794", "end": "2020-07-15 05:12:11.924244", "rc": 0, "start": "2020-07-15 05:12:11.522450", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}

2020-07-15 05:12:12.542972 Step: [Status]: Checking whether application containers are in running state 
task path: /utils/common/status_app_pod.yml:9
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pod -n shubham -l run=nginx --no-headers -o jsonpath='{.items[*].status.containerStatuses[*].ready}' | tr ' ' '\\n' | uniq", "delta": "0:00:00.421411", "end": "2020-07-15 05:12:12.525228", "rc": 0, "start": "2020-07-15 05:12:12.103817", "stderr": "", "stderr_lines": [], "stdout": "true", "stdout_lines": ["true"]}

2020-07-15 05:12:12.636693 Step: set_fact 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:133
details => {"ansible_facts": {"flag": "Pass"}, "changed": false}

2020-07-15 05:12:12.681128 Step: [Result]: Getting the final result of kubelet-service-kill experiment 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:136
details => {
    "msg": "kubelet-service-kill experiment has been Passed"
}
included: /utils/runtime/getting_failure_step.yml for 127.0.0.1

2020-07-15 05:12:12.992311 Step: [Failure-Detection]: Recording the name of failed step 
task path: /utils/runtime/getting_failure_step.yml:44
details => {"ansible_facts": {"failStep": "N/A"}, "changed": false}
included: /utils/runtime/update_chaos_result_resource.yml for 127.0.0.1

2020-07-15 05:12:13.863396 Step: [Result]: Update the chaos result CR to reflect EOT (End of Test) 
task path: /utils/runtime/update_chaos_result_resource.yml:23
details => {"changed": true, "checksum": "80dc5a38f7427593cafca41558fa0c0b6ed1ab16", "dest": "/tmp/chaos-result.yml", "gid": 0, "group": "root", "md5sum": "2724798210e50131f509c2a3772ebd12", "mode": "0644", "owner": "root", "size": 295, "src": "/tmp/ansible-tmp-1594789933.21-215507447172332/source", "state": "file", "uid": 0}

2020-07-15 05:12:15.727295 Step: [The END]: Apply the chaos result CR for kubelet-service-kill experiment 
task path: /utils/runtime/update_chaos_result_resource.yml:33
details => {"changed": true, "cmd": "kubectl apply -f /tmp/chaos-result.yml -n shubham", "delta": "0:00:01.651597", "end": "2020-07-15 05:12:15.681160", "failed_when_result": false, "rc": 0, "start": "2020-07-15 05:12:14.029563", "stderr": "", "stderr_lines": [], "stdout": "chaosresult.litmuschaos.io/kubelet-service-kill configured", "stdout_lines": ["chaosresult.litmuschaos.io/kubelet-service-kill configured"]}
META: ran handlers
META: ran handlers

2020-07-15 05:12:15.738522 ************ RELAX, EXPERIMENT ENDS! ************ 
127.0.0.1                  : ok=37   changed=17   unreachable=0    failed=0