community.okd.k8s hangs indefinitely

[manas-suleman]

Hi,

I am using ansible within a python 3.8 virtual environment for over a year and it has worked just fine. Recently, I have started having issue with community.okd.k8s and kubernetes.core.k8s modules as they hang indefinitely on playbook run. I think this issue got triggered with me updating ansible to the latest version, which updated ansible-core to the latest and latest ansible-core may be the root cause behind this. I have tried these:

1. reinstalling/updating community.okd and kubernetes.core collections
2. using 3.10 virtual environments
3. using pipx environments instead

I've shared detailed troubleshooting steps in a post on ansible forum here: https://forum.ansible.com/t/community-okd-k8s-and-kubernetes-core-k8s-hang-get-stuck-indefinitely/4821?u=123232455

Please let me know if I need to paste all the details here as well. Any help is appreciated. Thanks,

[gravesm]

I have been unable to reproduce your problem using the latest versions of kubernetes.core and community.okd on python 3.10 with ansible-core versions 2.14, 2.15, 2.16 and even the upcoming 2.17. Looking at that other thread, I'm not sure what versions of things you are currently running where it's failing for you. Could you clarify what your current set up is, and post the output with -vvvvvv for the failing task?

[manas-suleman]

Hi @gravesm, thank you for looking into this. Looks like it may have something to do with my setup. Here are the complete details:

I'm using Ubuntu 20.04.6 LTS on WSL2 on windows. Could it be because of WSL2? All the packages have been updated to the latest with sudo apt upgrade. I'm using a python3.10.14 virtual environment for ansible.

Here are the kubernetes.core and community.okd version details:

/home/user1/.ansible/collections/ansible_collections
Collection      Version
--------------- -------
community.okd 3.0.1
kubernetes.core 3.0.1
/home/user1/ansible_project/.venv/lib/python3.10/site-packages/ansible_collections
Collection      Version
--------------- -------
community.okd 2.3.0
kubernetes.core 2.4.2

Versions of kubernetes.core dependency packages:

jsonpatch                 1.33
kubernetes                29.0.0
PyYAML                    6.0.1

Here's the playbook I'm trying to run:

- name: Setting up OKD
  become: true
  become_user: root
  hosts: localhost
  tasks:
    - name: Login to OKD
      community.okd.k8s_auth:
        host: https://api.okd.example.com:6443
        password: "{{ okd_admin_password }}"
        username: "{{ okd_admin_user }}"
        validate_certs: false
      register: okd_auth
      environment:
        K8S_AUTH_API_KEY: okd_auth.openshift_auth

    - name: Create a k8s namespace
      community.okd.k8s:
        api_key: okd_auth.openshift_auth
        name: testing
        api_version: v1
        kind: Namespace
        state: present

Here's the command I'm using to run, ansible-core version and the verbose output: ansible-playbook --ask-vault-pass -i ansible/inventory.yml ansible/playbooks/okd_deploy.yml -vvvvvvvv

ansible-playbook [core 2.16.5]
  config file = None
  configured module search path = ['/home/user1/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/user1/ansible_project/.venv/lib/python3.10/site-packages/ansible
  ansible collection location = /home/user1/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/user1/ansible_project/.venv/bin/ansible-playbook
  python version = 3.10.14 (main, Apr  6 2024, 18:45:05) [GCC 9.4.0] (/home/user1/ansible_project/.venv/bin/python3)
  jinja version = 3.1.3
  libyaml = True

TASK [Create a k8s namespace] ****************************************************************************************************************************************************************************************************************
task path: /home/user1/ansible_project/ansible/playbooks/okd_deploy.yml:20
redirecting (type: action) community.okd.k8s to kubernetes.core.k8s_info
redirecting (type: action) community.okd.k8s to kubernetes.core.k8s_info
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: user1
<localhost> EXEC /bin/sh -c 'echo ~user1 && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user1/.ansible/tmp `"&& mkdir "` echo /home/user1/.ansible/tmp/ansible-tmp-1713014858.7723665-7103-240287913627696 `" && echo ansible-tmp-1713014858.7723665-7103-240287913627696="` echo /home/user1/.ansible/tmp/ansible-tmp-1713014858.7723665-7103-240287913627696 `" ) && sleep 0'
Including module_utils file ansible/__init__.py
Including module_utils file ansible/module_utils/__init__.py
Including module_utils file ansible/module_utils/basic.py
Including module_utils file ansible/module_utils/_text.py
Including module_utils file ansible/module_utils/common/_json_compat.py
Including module_utils file ansible/module_utils/common/__init__.py
Including module_utils file ansible/module_utils/common/_utils.py
Including module_utils file ansible/module_utils/common/arg_spec.py
Including module_utils file ansible/module_utils/common/file.py
Including module_utils file ansible/module_utils/common/locale.py
Including module_utils file ansible/module_utils/common/parameters.py
Including module_utils file ansible/module_utils/common/collections.py
Including module_utils file ansible/module_utils/common/process.py
Including module_utils file ansible/module_utils/common/sys_info.py
Including module_utils file ansible/module_utils/common/text/converters.py
Including module_utils file ansible/module_utils/common/text/__init__.py
Including module_utils file ansible/module_utils/common/text/formatters.py
Including module_utils file ansible/module_utils/common/validation.py
Including module_utils file ansible/module_utils/common/warnings.py
Including module_utils file ansible/module_utils/compat/selectors.py
Including module_utils file ansible/module_utils/compat/__init__.py
Including module_utils file ansible/module_utils/compat/_selectors2.py
Including module_utils file ansible/module_utils/compat/selinux.py
Including module_utils file ansible/module_utils/distro/__init__.py
Including module_utils file ansible/module_utils/distro/_distro.py
Including module_utils file ansible/module_utils/errors.py
Including module_utils file ansible/module_utils/parsing/convert_bool.py
Including module_utils file ansible/module_utils/parsing/__init__.py
Including module_utils file ansible/module_utils/pycompat24.py
Including module_utils file ansible/module_utils/six/__init__.py
Including module_utils file ansible_collections/community/okd/plugins/module_utils/k8s.py
Including module_utils file ansible_collections/__init__.py
Including module_utils file ansible_collections/community/__init__.py
Including module_utils file ansible_collections/community/okd/__init__.py
Including module_utils file ansible_collections/community/okd/plugins/__init__.py
Including module_utils file ansible_collections/community/okd/plugins/module_utils/__init__.py
Including module_utils file ansible_collections/community/okd/plugins/module_utils/openshift_common.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/args_common.py
Including module_utils file ansible_collections/kubernetes/__init__.py
Including module_utils file ansible_collections/kubernetes/core/__init__.py
Including module_utils file ansible_collections/kubernetes/core/plugins/__init__.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/__init__.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/client/__init__.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/client/resource.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/__init__.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/core.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/exceptions.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/resource.py
Including module_utils file ansible/module_utils/urls.py
Including module_utils file ansible/module_utils/compat/typing.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/runner.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py
Including module_utils file ansible/module_utils/common/dict_transformations.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/apply.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/common.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/exceptions.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/hashes.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/waiter.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8sdynamicclient.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/selector.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/version.py
Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/_version.py
Using module file /home/user1/.ansible/collections/ansible_collections/community/okd/plugins/modules/k8s.py
<localhost> PUT /home/user1/.ansible/tmp/ansible-local-6898vaf7ji1n/tmpqv1091oq TO /home/user1/.ansible/tmp/ansible-tmp-1713014858.7723665-7103-240287913627696/AnsiballZ_k8s.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/user1/.ansible/tmp/ansible-tmp-1713014858.7723665-7103-240287913627696/ /home/user1/.ansible/tmp/ansible-tmp-1713014858.7723665-7103-240287913627696/AnsiballZ_k8s.py && sleep 0'
<localhost> EXEC /bin/sh -c 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-bvzyahmxgwhlgfnbgyiuwradxlofjlzt ; /usr/bin/python3 /home/user1/.ansible/tmp/ansible-tmp-1713014858.7723665-7103-240287913627696/AnsiballZ_k8s.py'"'"' && sleep 0'

Apologies for the long output but I wanted to include as much detail as possible. In the verbose output above, community.okd.k8s redirects to kubernetes.core.k8s_info which is a read-only module. Could that be a problem?

Really appreciate you getting back to me on this.

[gravesm]

I just realized you have a few problems with your playbook. The beginning of your second task should look like:

    - name: Create a k8s namespace
      community.okd.k8s:
        api_key: "{{ okd_auth.openshift_auth.api_key }}"

Note that what you currently have is passing the literal string "okd_auth.openshift_auth" as the API key. Also, okd_auth.openshift_auth is a dictionary, and you need the api_key property on it. Depending on your environment, you may also need to specify the host and validate_certs on that task.

I still don't know why it's hanging instead of just returning an error message, but maybe this will fix your problem.

[manas-suleman]

Hi,

Thank you for the comment. That api_key error was certainly an error I needed to fix but it wasn't the root cause. The root cause was me not setting up the environment properly. I've now added the ansible_python_interpreter variable to point to python binary inside the virtual environment, removed all the 'supporting python packages' that were installed outside of the virtual environment and that basically fixed it for me.

There were some python packages that were not installed within the environment and ansible was picking them from outside the environment. And that caused the issue apparently.

Apologies for taking up your time and thank you for your support.

[gravesm]

Glad to hear it's sorted.