k8s module fails on FIPS enabled OpenShift clusters

[fbladilo]

SUMMARY

Attempting to run k8s ansible module in a FIPS enabled OpenShift cluster fails with :

An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ValueError'>, original message: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

We are using the k8s module extensively with Operator SDK ansible, currently requirements.yml in Operator SDK detailing versions of collections :

  - name: community.kubernetes
    version: "2.0.1"
  - name: operator_sdk.util
    version: "0.4.0"
  - name: kubernetes.core
    version: "2.3.1"
  - name: cloud.common
    version: "2.1.1"

ISSUE TYPE

• Bug Report

COMPONENT NAME

k8s

ANSIBLE VERSION

sh-4.4$ ansible --version
ansible 2.9.27
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]

COLLECTION VERSION

  - name: community.kubernetes
    version: "2.0.1"

CONFIGURATION

sh-4.4$ ansible-config dump --only-changed
DEFAULT_GATHERING(env: ANSIBLE_GATHERING) = explicit
DEFAULT_MODULE_PATH(/etc/ansible/ansible.cfg) = ['/usr/share/ansible/openshift']
DEFAULT_ROLES_PATH(/etc/ansible/ansible.cfg) = ['/opt/ansible/roles']

OS / ENVIRONMENT

Operator SDK image used (RHEL8.6 based) : quay.io/operator-framework/ansible-operator:v1.22.0 Openshift Version : 4.10

STEPS TO REPRODUCE

1. Create an ansible operator with Operator SDK
2. Deploy ansible operator in an OpenShift cluster with FIPS enabled
3. Call k8s ansible module within the operator role/playbook

Alternatively , you can just run the playbook pasted below on a FIPS enabled system.

sh-4.4$ cat test.yml 
- hosts: localhost
  tasks:
  - name: "Load cluster API groups"
    set_fact:
      api_groups: "{{ lookup('k8s', cluster_info='api_groups') }}"

EXPECTED RESULTS

Run k8s successfully in FIPS enabled systems

ACTUAL RESULTS

sh-4.4$ ansible-playbook -vvvvvv test.yml 
ansible-playbook 2.9.27
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
Set default localhost to localhost
Parsed /etc/ansible/hosts inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.8/site-packages/ansible/plugins/callback/default.py
Attempting to use 'actionable' callback.
Skipping callback 'actionable', as we already have a stdout callback.
Attempting to use 'aws_resource_actions' callback.
Attempting to use 'cgroup_memory_recap' callback.
Attempting to use 'cgroup_perf_recap' callback.
Attempting to use 'context_demo' callback.
Attempting to use 'counter_enabled' callback.
Skipping callback 'counter_enabled', as we already have a stdout callback.
Attempting to use 'debug' callback.
Skipping callback 'debug', as we already have a stdout callback.
Attempting to use 'dense' callback.
Skipping callback 'dense', as we already have a stdout callback.
Attempting to use 'dense' callback.
Skipping callback 'dense', as we already have a stdout callback.
Attempting to use 'foreman' callback.
Attempting to use 'full_skip' callback.
Skipping callback 'full_skip', as we already have a stdout callback.
Attempting to use 'grafana_annotations' callback.
Attempting to use 'hipchat' callback.
Attempting to use 'jabber' callback.
Attempting to use 'json' callback.
Skipping callback 'json', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'log_plays' callback.
Attempting to use 'logdna' callback.
Attempting to use 'logentries' callback.
Attempting to use 'logstash' callback.
Attempting to use 'mail' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'nrdp' callback.
Attempting to use 'null' callback.
Skipping callback 'null', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'osx_say' callback.
Attempting to use 'profile_roles' callback.
Attempting to use 'profile_tasks' callback.
Attempting to use 'say' callback.
Attempting to use 'selective' callback.
Skipping callback 'selective', as we already have a stdout callback.
Attempting to use 'skippy' callback.
Skipping callback 'skippy', as we already have a stdout callback.
Attempting to use 'slack' callback.
Attempting to use 'splunk' callback.
Attempting to use 'stderr' callback.
Skipping callback 'stderr', as we already have a stdout callback.
Attempting to use 'sumologic' callback.
Attempting to use 'syslog_json' callback.
Attempting to use 'timer' callback.
Attempting to use 'tree' callback.
Attempting to use 'unixy' callback.
Skipping callback 'unixy', as we already have a stdout callback.
Attempting to use 'yaml' callback.
Skipping callback 'yaml', as we already have a stdout callback.

PLAYBOOK: test.yml *******************************************************************************************************************************
Positional arguments: test.yml
verbosity: 6
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/etc/ansible/hosts',)
forks: 5
1 plays in test.yml

PLAY [localhost] *********************************************************************************************************************************
META: ran handlers

TASK [Load cluster API groups] *******************************************************************************************************************
task path: /tmp/test.yml:3
fatal: [localhost]: FAILED! => {
    "msg": "An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ValueError'>, original message: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS"
}

PLAY RECAP ***************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

sh-4.4$ 

[Akasurde]

@fbladilo Thanks for reporting this issue. Could you please provide the output for -vvv so that we can see the full stack trace without which it is difficult to debug the issue?

[fbladilo]

@Akasurde I added the ansible debug output as requested to the "actual results" issue section. I was able to trigger easily with a playbook like this :

sh-4.4$ cat test.yml 
- hosts: localhost
  tasks:
  - name: "Load cluster API groups"
    set_fact:
      api_groups: "{{ lookup('k8s', cluster_info='api_groups') }}"
  - debug:
      var: api_groups

[gravesm]

@fbladilo Sorry for the delay. Could you try using the fully qualified collection name for the lookup plugin? So:

api_groups: "{{ lookup('kubernetes.core.k8s', cluster_info='api_groups') }}"

[ozekidesu]

I ran into this issue today. The module is using and md5 hash which is not FIPS 140-2 osrcp-{0}.json'.format(hashlib.md5(default_cache_id).hexdigest()

The full traceback is:
  File "/tmp/ansible_kubernetes.core.k8s_payload_5cdslgzl/ansible_kubernetes.core.k8s_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/common.py", line 294, in get_api_client
    return DynamicClient(kubernetes.client.ApiClient(configuration))
  File "/home/user/.local/lib/python3.6/site-packages/openshift/dynamic/client.py", line 40, in __init__
    K8sDynamicClient.__init__(self, client, cache_file=cache_file, discoverer=discoverer)
  File "/home/user/.local/lib/python3.6/site-packages/kubernetes/dynamic/client.py", line 84, in __init__
    self.__discoverer = discoverer(self, cache_file)
  File "/home/user/.local/lib/python3.6/site-packages/kubernetes/dynamic/discovery.py", line 224, in __init__
    Discoverer.__init__(self, client, cache_file)
  File "/home/user/.local/lib/python3.6/site-packages/kubernetes/dynamic/discovery.py", line 48, in __init__
    default_cachefile_name = 'osrcp-{0}.json'.format(hashlib.md5(default_cache_id).hexdigest())
fatal: [ip-10-1-2-3.us-east-1.compute.internal]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "api_key": null,
            "api_version": "v1",
            "append_hash": false,
            "apply": false,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "context": null,
            "delete_options": null,
            "force": false,
            "host": null,
            "kind": "Namespace",
            "kubeconfig": null,
            "merge_type": null,
            "name": "bigbang",
            "namespace": null,
            "password": null,
            "persist_config": null,
            "proxy": null,
            "resource_definition": null,
            "src": null,
            "state": "present",
            "template": null,
            "username": null,
            "validate": null,
            "validate_certs": null,
            "wait": false,
Failed to get client due to [digital envelope routines: EVP_DigestInit_ex] disabled for fips())

Per FIPS 104-2 Annex A Approved Security Functions for FIPS PUB 140-2 the following Secure Hash Standard (SHS) (SHA-1, SHA-224, SHA-256, SHA-384, SHA-512, SHA-512/224 and SHA-512/256) are compliant.

[gravesm]

@ozekidesu The kubernetes.core collection has used SHA 256 since version 2.0. You don't specify what version of Ansible or what version of the collection you are using, but I would suggest upgrading and making sure you are using the fully qualified collection name.