Switch to using k8s_exec module instead of shell module + kubectl

geerlingguy commented 4 years ago

In #5, I discovered the k8s_exec module from this PR (https://github.com/ansible/ansible/pull/55029) does not work when running inside an Ansible-based Operator due to some proxy request handling the Operator does for Kubernetes API requests.

The gist of the problem is k8s_exec uses a websocket to communicate with Kubernetes to run an exec command, but the proxy does not handle the 101 handshake response correctly (instead returning a 200), which results in a failure of the k8s_exec module.

I was going to try to get that issue fixed in #5, but as a workaround, I'm currently using kubectl, which is installed in the operator image with the following line:

# Install kubectl.
COPY --from=lachlanevenson/k8s-kubectl:v1.16.2 /usr/local/bin/kubectl /usr/local/bin/kubectl

This is a little fragile, as it means the kubectl currently shipping with this operator is locked into a specific version (which likely won't cause issues, but isn't wonderful especially if it could be used as an attack vector if a vulnerability is found with whatever the current version is).

So for this issue to be complete, the following should be done:

[ ] Add back roles/tower/library/k8s_exec.py based on this PR.
[ ] Switch each of the tasks in the role's main.yml to use k8s_exec instead of shell + kubectl.
[ ] Work to resolve upstream issue https://github.com/operator-framework/operator-sdk/issues/2204
[ ] Verify the switch works correctly when running inside the operator, and doesn't throw errors.

geerlingguy commented 4 years ago

Working on testing this out again in a new branch (I'll push it up shortly) so I can post an update to https://github.com/operator-framework/operator-sdk/issues/2204

geerlingguy commented 4 years ago

Also need to complete https://github.com/geerlingguy/tower-operator/issues/18 first, just to make sure I'm on the latest version of the operator for testing purposes.

geerlingguy commented 4 years ago

Still getting:

raise ApiException(status=0, reason=str(e))
kubernetes.client.rest.ApiException: (0)
Reason: Handshake status 200 OK

Using base image version v0.12.0, as well as v0.14.0

geerlingguy commented 4 years ago

Steps to reproduce:

$ git checkout k8s_exec
$ minikube start --memory 6g --cpus 4
$ molecule test -s test-minikube

# while that's running, when you get to reconciliation, in another terminal, run:
$ kubectl logs -f -l name=tower-operator -c ansible

The logs will end (after the first failed operator playbook run) with something like:

TASK [tower : Migrate the database if the K8s resources were updated.] *********
task path: /opt/ansible/roles/tower/tasks/main.yml:50
.modules.k8s_exec', init_globals=None, run_name='__main__', alter_sys=True)
  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code
    exec(code, run_globals)
  File \"/tmp/ansible_k8s_exec_payload_kzp9svbw/ansible_k8s_exec_payload.zip/ansible/modules/k8s_exec.py\", line 154, in <module>
  File \"/tmp/ansible_k8s_exec_payload_kzp9svbw/ansible_k8s_exec_payload.zip/ansible/modules/k8s_exec.py\", line 141, in main
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 32, in stream
    return func(*args, **kwargs)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py\", line 835, in connect_get_namespaced_pod_exec
    (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py\", line 935, in connect_get_namespaced_pod_exec_with_http_info
    collection_formats=collection_formats)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 321, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 155, in __call_api
    _request_timeout=_request_timeout)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 27, in _intercept_request_call
    return ws_client.websocket_call(config, *args, **kwargs)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 255, in websocket_call
    raise ApiException(status=0, reason=str(e))
kubernetes.client.rest.ApiException: (0)
Reason: Handshake status 200 OK

", "mofatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "/usr/local/lib/python3.6/site-packages/kubernetes/config/kube_config.py:509: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config_dict=yaml.load(f),
Traceback (most recent call last):
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 249, in websocket_call
    client = WSClient(configuration, get_websocket_url(url), headers)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 72, in __init__
    self.sock.connect(url, header=header)
  File \"/usr/local/lib/python3.6/site-packages/websocket/_core.py\", line 226, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File \"/usr/local/lib/python3.6/site-packages/websocket/_handshake.py\", line 80, in handshake
    status, resp = _get_resp_headers(sock)
  File \"/usr/local/lib/python3.6/site-packages/websocket/_handshake.py\", line 165, in _get_resp_headers
    raise WebSocketBadStatusException(\"Handshake status %d %s\", status, status_message, resp_headers)
websocket._exceptions.WebSocketBadStatusException: Handshake status 200 OK

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1579206499.4669755-175197898839380/AnsiballZ_k8s_exec.py\", line 102, in <module>
    _ansiballz_main()
  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1579206499.4669755-175197898839380/AnsiballZ_k8s_exec.py\", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1579206499.4669755-175197898839380/AnsiballZ_k8s_exec.py\", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.k8s_exec', init_globals=None, run_name='__main__', alter_sys=True)
  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code
    exec(code, run_globals)
  File \"/tmp/ansible_k8s_exec_payload_kzp9svbw/ansible_k8s_exec_payload.zip/ansible/modules/k8s_exec.py\", line 154, in <module>
  File \"/tmp/ansible_k8s_exec_payload_kzp9svbw/ansible_k8s_exec_payload.zip/ansible/modules/k8s_exec.py\", line 141, in main
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 32, in stream
    return func(*args, **kwargs)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py\", line 835, in connect_get_namespaced_pod_exec
    (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py\", line 935, in connect_get_namespaced_pod_exec_with_http_info
    collection_formats=collection_formats)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 321, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py\", line 155, in __call_api
    _request_timeout=_request_timeout)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/stream.py\", line 27, in _intercept_request_call
    return ws_client.websocket_call(config, *args, **kwargs)
  File \"/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py\", line 255, in websocket_call
    raise ApiException(status=0, reason=str(e))
kubernetes.client.rest.ApiException: (0)
Reason: Handshake status 200 OK

", "module_stdout": "", "msg": "MODULE FAILURE
See stdout/stderr for the exact error", "rc": 1}

geerlingguy commented 4 years ago

Updated the upstream issue. I'm going to do some digging in the proxy that's set up in the operator and see if there's an easy fix or not.

stale[bot] commented 4 years ago

This issue has been marked 'stale' due to lack of recent activity. If there is no further activity, the issue will be closed in another 30 days. Thank you for your contribution!

Please read this blog post to see the reasons why I mark issues as stale.

stale[bot] commented 4 years ago

This issue is no longer marked for closure.

geerlingguy commented 4 years ago

This should be fixable now—see https://github.com/operator-framework/operator-sdk/issues/2204 and PR https://github.com/operator-framework/operator-sdk/issues/2204

geerlingguy commented 4 years ago

Now that the operator is being maintained in the Ansible namespace, I am not going to maintain this project anymore (as I consider it a historic artifact which was used to as a base for the ansible-namespace version, and I'm redirecting all issues and PRs to the Ansible-maintained version: https://github.com/ansible/awx-operator

Moved this issue to: https://github.com/ansible/awx-operator/issues/3

geerlingguy / tower-operator

Switch to using k8s_exec module instead of shell module + kubectl #8