Inline script step fails on Kubernetes pod node

[hdimitriou]

When I try to run a 'script' node step on a pod I get

DEBUG: kubernetes-plugin: config file
--
18:42:08 | DEBUG: kubernetes-plugin: /home/rundeck/.kube/config-rd-K1
18:42:08 | DEBUG: kubernetes-plugin: -------------------
18:42:08 | DEBUG: kubernetes-plugin: getting settings from file /home/rundeck/.kube/config-rd-K1
18:42:08 | DEBUG: kubernetes-model-source: --------------------------
18:42:08 | DEBUG: kubernetes-model-source: Pod Name:  rundeck-worker-replicaset-c57jv
18:42:08 | DEBUG: kubernetes-model-source: Namespace: rundeck-workers
18:42:08 | DEBUG: kubernetes-model-source: Container: rundeck-worker
18:42:08 | DEBUG: kubernetes-model-source: --------------------------
18:42:08 | DEBUG: kubernetes-model-source: Copying file from /home/rundeck/var/tmp/dispatch2691833068787593122.tmp to /tmp/72-77-rundeck-worker-replicaset-c57jv-rundeck-worker-dispatch-script.tmp.sh
18:42:08 | Traceback (most recent call last):
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/kubernetes/stream/ws_client.py", line 249, in websocket_call
18:42:08 | client = WSClient(configuration, get_websocket_url(url), headers)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/kubernetes/stream/ws_client.py", line 72, in __init__
18:42:08 | self.sock.connect(url, header=header)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/websocket/_core.py", line 226, in connect
18:42:08 | self.handshake_response = handshake(self.sock, *addrs, **options)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/websocket/_handshake.py", line 79, in handshake
18:42:08 | status, resp = _get_resp_headers(sock)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/websocket/_handshake.py", line 160, in _get_resp_headers
18:42:08 | raise WebSocketBadStatusException("Handshake status %d %s", status, status_message, resp_headers)
18:42:08 | websocket._exceptions.WebSocketBadStatusException: Handshake status 403 Forbidden
18:42:08 |  
18:42:08 | During handling of the above exception, another exception occurred:
18:42:08 |  
18:42:08 | Traceback (most recent call last):
18:42:08 | File "/home/rundeck/libext/cache/kubernetes-plugin-1.10.1-SNAPSHOT/pods-copy-file.py", line 69, in <module>
18:42:08 | main()
18:42:08 | File "/home/rundeck/libext/cache/kubernetes-plugin-1.10.1-SNAPSHOT/pods-copy-file.py", line 65, in main
18:42:08 | common.copy_file(name, container, source_file, destination_path, destination_file_name)
18:42:08 | File "/home/rundeck/libext/cache/kubernetes-plugin-1.10.1-SNAPSHOT/common.py", line 374, in copy_file
18:42:08 | _preload_content=False)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/kubernetes/stream/stream.py", line 32, in stream
18:42:08 | return func(*args, **kwargs)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/apis/core_v1_api.py", line 835, in connect_get_namespaced_pod_exec
18:42:08 | (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/apis/core_v1_api.py", line 935, in connect_get_namespaced_pod_exec_with_http_info
18:42:08 | collection_formats=collection_formats)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/api_client.py", line 321, in call_api
18:42:08 | _return_http_data_only, collection_formats, _preload_content, _request_timeout)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/api_client.py", line 155, in __call_api
18:42:08 | _request_timeout=_request_timeout)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/kubernetes/stream/stream.py", line 27, in _intercept_request_call
18:42:08 | return ws_client.websocket_call(config, *args, **kwargs)
18:42:08 | File "/usr/local/lib/python3.5/dist-packages/kubernetes/stream/ws_client.py", line 255, in websocket_call
18:42:08 | raise ApiException(status=0, reason=str(e))
18:42:08 | kubernetes.client.rest.ApiException: (0)
18:42:08 | Reason: Handshake status 403 Forbidden

'Command' node steps work fine on those pods with the same settings. This is just another step on the same job.

I have setup the project to use "Kubernetes / Pods / File Copier" with the very same setting I have for 'Kubernetes / Pods / Node Executor'

[hdimitriou]

I also tried 'Kubernetes / Pods / Execution Script' but that didn't work either

18:59:49 | DEBUG: kubernetes-plugin: config file
-- | --
18:59:49 | DEBUG: kubernetes-plugin: /home/rundeck/.kube/config-rd-K1
18:59:49 | DEBUG: kubernetes-plugin: -------------------
18:59:49 | DEBUG: kubernetes-plugin: getting settings from file /home/rundeck/.kube/config-rd-K1
18:59:49 | DEBUG: kubernetes-model-source: --------------------------
18:59:49 | DEBUG: kubernetes-model-source: Pod Name:  rundeck-worker-replicaset-c57jv
18:59:49 | DEBUG: kubernetes-model-source: Namespace: rundeck-workers
18:59:49 | DEBUG: kubernetes-model-source: --------------------------
18:59:49 | Traceback (most recent call last):
18:59:49 | File "/home/rundeck/libext/cache/kubernetes-plugin-1.10.1-SNAPSHOT/pods-run-script.py", line 156, in <module>
18:59:49 | main()
18:59:49 | File "/home/rundeck/libext/cache/kubernetes-plugin-1.10.1-SNAPSHOT/pods-run-script.py", line 79, in main
18:59:49 | temp.write(script)
18:59:49 | File "/usr/lib/python3.5/tempfile.py", line 622, in func_wrapper
18:59:49 | return func(*args, **kwargs)
18:59:49 | TypeError: a bytes-like object is required, not 'str'
18:59:50 | [Kubernetes-InlineScript-Step]: result code: 1
18:59:50 | Failed: NonZeroResultCode: Script result code was: 1

[hdimitriou]

Ok so I figured this out, first of all Reason: Handshake status 403 Forbidden was due to the fact I had wrong ServiceAccount permission

Still however there is a bug on the code

kubernetes.client.rest.ApiException: (0)
Reason: Handshake status 404 Not Found

and this has to do forgetting to pass the namespace variable to the copy_file function from the pods-copy-file.py

[hdimitriou]

I created this PR https://github.com/rundeck-plugins/kubernetes/pull/40 but i dont know how to link it

[ltamaster]

Thanks @hdimitriou , merged

[hdimitriou]

PR is merged so I am closing this