Unable to create AWX Instance: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

andrewvillano commented 1 year ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

I am at the point here: https://github.com/ansible/awx-operator

Where I:" Finally, run kustomize again to create the AWX instance in your cluster:"

However, I receive the following error message: TASK [Get information about the cluster] **** fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ValueError'>, original message: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS"}

AWX Operator version

1.0.0

AWX version

?

Kubernetes platform

minikube

Kubernetes/Platform version

1.28.0

Modifications

no

Steps to reproduce

Follow steps here: https://github.com/ansible/awx-operator

leading up to : ./kustomize build . | kubectl apply -f -

Expected results

Environment to build

Actual results

Error message received

TASK [Get information about the cluster] **** fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ValueError'>, original message: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS"}

Additional information

No response

Operator Logs

No response

djyasin commented 1 year ago

Hello, this appears to be less of a bug report or feature request and more of a question. Could you please ask this on our mailing list? See https://github.com/ansible/awx/#get-involved for information for ways to connect with us.

shanemcd commented 1 year ago

I think this is a bug report, just not with our code. I would encourage you to go file this over in https://github.com/ansible-collections/kubernetes.core

adamagnew commented 1 year ago

I narrowed down the error as beginning in awx_operator image 0.30.0 .. Still don't know why. The kubernetes.core issue that was opened believe that the problem should have gone away in kubernetes.core 2.0+ which each version of the awx operator appears to be well past by the requriements.yml.

ravitejb commented 1 year ago

@andrewvillano were you able to get the fix for this? I'm also getting the same error. my awx operator is quay.io/ansible/awx-operator:1.2.0 my collections requirements are as below

collections:
  - name: kubernetes.core
    version: '>=2.3.2'
  - name: operator_sdk.util
    version: "0.4.0"

ansbile version is

bash-4.4$ ansible --version
ansible 2.9.27
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/usr/share/ansible/openshift']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.13 (default, Jun 14 2022, 17:49:07) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]

OS id RHEL8 with FIPS enabled k8s cluster

andrewvillano commented 1 year ago

Still having the same issue

Mr. Andrew V. Villano, CCNA, RHCSA Linux Administrator United States District Court Theodore Roosevelt United States Courthouse 225 Cadman Plaza East, Room 304S Brooklyn, NY 11201-1818 (718) 613 - 2682 @.***

From: Ravi Teja @.> Sent: Thursday, February 23, 2023 5:41:05 PM To: ansible/awx-operator @.> Cc: Andrew Villano @.>; Mention @.> Subject: Re: [ansible/awx-operator] Unable to create AWX Instance: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS (Issue #1122)

CAUTION - EXTERNAL:

@andrewvillanohttps://github.com/andrewvillano were you able to get the fix for this? I'm also getting the same error. my awx operator is quay.io/ansible/awx-operator:1.2.0 my collections requirements are as below

collections:

name: kubernetes.core version: '>=2.3.2'
name: operator_sdk.util version: "0.4.0"

ansbile version is

bash-4.4$ ansible --version ansible 2.9.27 config file = /etc/ansible/ansible.cfg configured module search path = ['/usr/share/ansible/openshift'] ansible python module location = /usr/local/lib/python3.8/site-packages/ansible executable location = /usr/local/bin/ansible python version = 3.8.13 (default, Jun 14 2022, 17:49:07) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]

OS id RHEL8 with FIPS enabled k8s cluster

— Reply to this email directly, view it on GitHubhttps://github.com/ansible/awx-operator/issues/1122#issuecomment-1442525662, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFY26XTVHDMHS2WCCWUFQMDWY7RQDANCNFSM6AAAAAASAHGJ7I. You are receiving this because you were mentioned.Message ID: @.***>

CAUTION - EXTERNAL EMAIL: This email originated outside the Judiciary. Exercise caution when opening attachments or clicking on links.

kurokobo commented 1 year ago

@djyasin @shanemcd Not tested yet since I don't have FIPS-enabled node but I think FQCN should be used here:

https://github.com/ansible/awx-operator/blob/b5f255cf008472927bb9af1b3ef8235d3967bd05/roles/common/tasks/main.yml#L5

There are two k8s lookup plugin inside the operator image and seems newer plugin from kubernetes.core never be used. If this issue came from old plugin (again, not tested on my side), this issue should be re-opened and should be fixed on Operater side.

$ kubectl -n awx exec -it deployment/awx-operator-controller-manager -- bash

# "k8s" refers bundled (old) plugin
bash-4.4$ ansible-doc -t lookup k8s | head -n 1
> K8S    (/usr/local/lib/python3.8/site-packages/ansible/plugins/lookup/k8s.py)

# Updated 2021
bash-4.4$ ls -l /usr/local/lib/python3.8/site-packages/ansible/plugins/lookup/k8s.py
-rw-r--r--. 1 root root 11018 Oct 11  2021 /usr/local/lib/python3.8/site-packages/ansible/plugins/lookup/k8s.py

# "kubernetes.core.k8s" refers plugin from collection
bash-4.4$ ansible-doc -t lookup kubernetes.core.k8s | head -n 1
> K8S    (/opt/ansible/.ansible/collections/ansible_collections/kubernetes/core/plugins/lookup/k8s.py)

kurokobo commented 1 year ago

@djyasin @shanemcd I've confirmed that this issue came from old lookup plugin. This should be re-opened.

# Ensure the node is in FIPS mode
$ sudo fips-mode-setup --check
FIPS mode is enabled.

# Deploy AWX Operator 1.2.0
$ cd ~
$ git clone https://github.com/ansible/awx-operator.git
$ cd awx-operator
$ git checkout 1.2.0
$ export NAMESPACE=awx
$ make deploy

# Dig into the Operator
$ kubectl -n awx exec -it deployment/awx-operator-controller-manager -- bash

# Create playbooks to test both plugins
bash-4.4$ cd /tmp
## Test bundled plugin
bash-4.4$ cat <<EOF > test_bundled_k8s.yml
- hosts: localhost
  tasks:
  - set_fact:
      api_groups: "{{ lookup('k8s', cluster_info='api_groups') }}"     👈👈👈 As current implementation
EOF
## Test plugin from collection
bash-4.4$ cat <<EOF > test_collection_k8s.yml
- hosts: localhost
  tasks:
  - set_fact:
      api_groups: "{{ lookup('kubernetes.core.k8s', cluster_info='api_groups') }}"     👈👈👈 FQCN
EOF

The task with old plugin fails as described in this issue, and the plugin from newer collections works well.

# Bundled plugin causes error as this issue
bash-4.4$ ansible-playbook test_bundled_k8s.yml

PLAY [localhost] *******************************************************************************************************

TASK [set_fact] ********************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'k8s'. Error was a <class 'ValueError'>, original message: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS"}

PLAY RECAP *************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

# Collection plugin works as expected
bash-4.4$ ansible-playbook test_collection_k8s.yml

PLAY [localhost] *******************************************************************************************************

TASK [set_fact] ********************************************************************************************************
ok: [localhost]

PLAY RECAP *************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

On the node not in FIPS mode, I can confirm that both plugins work and return same list, so I think we can safely replace k8s with kubernetes.core.k8s.

$ sudo fips-mode-setup --check
Installation of FIPS modules is not completed.
FIPS mode is disabled.

$ kubectl -n awx exec -it deployment/awx-operator-controller-manager -- bash
bash-4.4$ cd /tmp
bash-4.4$ cat <<EOF > compare_k8s.yml
- hosts: localhost
  tasks:
  - debug:
      var: api_groups
    vars:
      api_groups:
        bundle: "{{ lookup('k8s', cluster_info='api_groups') }}"
        collection: "{{ lookup('kubernetes.core.k8s', cluster_info='api_groups') }}"
EOF

bash-4.4$ ansible-playbook compare_k8s.yml
bash-4.4$ ansible-playbook compare_k8s.yml

PLAY [localhost] *******************************************************************************************************

TASK [debug] ***********************************************************************************************************
ok: [localhost] => {
    "api_groups": {
        "bundle": [
            "",
            "apiregistration.k8s.io",
            "apps",
            "events.k8s.io",
            "authentication.k8s.io",
            "authorization.k8s.io",
            "autoscaling",
            "batch",
            "certificates.k8s.io",
            "networking.k8s.io",
            "policy",
            "rbac.authorization.k8s.io",
            "storage.k8s.io",
            "admissionregistration.k8s.io",
            "apiextensions.k8s.io",
            "scheduling.k8s.io",
            "coordination.k8s.io",
            "node.k8s.io",
            "discovery.k8s.io",
            "flowcontrol.apiserver.k8s.io",
            "helm.cattle.io",
            "k3s.cattle.io",
            "traefik.containo.us",
            "awx.ansible.com",
            "metrics.k8s.io"
        ],
        "collection": [
            "",
            "apiregistration.k8s.io",
            "apps",
            "events.k8s.io",
            "authentication.k8s.io",
            "authorization.k8s.io",
            "autoscaling",
            "batch",
            "certificates.k8s.io",
            "networking.k8s.io",
            "policy",
            "rbac.authorization.k8s.io",
            "storage.k8s.io",
            "admissionregistration.k8s.io",
            "apiextensions.k8s.io",
            "scheduling.k8s.io",
            "coordination.k8s.io",
            "node.k8s.io",
            "discovery.k8s.io",
            "flowcontrol.apiserver.k8s.io",
            "helm.cattle.io",
            "k3s.cattle.io",
            "traefik.containo.us",
            "awx.ansible.com",
            "metrics.k8s.io"
        ]
    }
}

PLAY RECAP *************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

shanemcd commented 1 year ago

As always, thanks for being awesome @kurokobo 🙂

kurokobo commented 1 year ago

After replacing k8s lookup plugin with FQCN, the task that failed in this issue (Get information about the cluster) completed with ok. However, another task causes the same issue. This came from operator_sdk.util.k8s_status module.

TASK [Update admin password status] ********************************
fatal: [localhost]: FAILED! => {"changed": false, "error": "[digital envelope routines: EVP_DigestInit_ex] disabled for FIPS", "msg": "Failed to get client due to %s"}

https://github.com/ansible/awx-operator/blob/b5f255cf008472927bb9af1b3ef8235d3967bd05/roles/installer/tasks/update_status.yml#L2-L9

According to the issue on operator-sdk repository (https://github.com/operator-framework/operator-sdk/issues/5723), old kubernetes python module causes this issue. In addition, I have noticed that newer kubernetes python module makes bundled (non-FQCN) k8s lookup plugin work (It is still best practice to use FQCN, though.).

So I think we have two choices to solve this issue:

A) Upgrade Operator SDK to 1.26.0 or later that contains fixed kuberenetes python module.
B) Upgrade only kubernetes python module to 25.3.0 or later (https://github.com/operator-framework/operator-sdk/issues/5723#issuecomment-1291586445)

This is the minimal patch for choice B. Not fully tested but AWX instance can be deployed by Operator with this patch on the node in FIPS mode:

diff --git a/Dockerfile b/Dockerfile
index 5022a55..99a4ef8 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -12,7 +12,8 @@ ENV DEFAULT_AWX_VERSION=${DEFAULT_AWX_VERSION}
 ENV OPERATOR_VERSION=${OPERATOR_VERSION}

 COPY requirements.yml ${HOME}/requirements.yml
-RUN ansible-galaxy collection install -r ${HOME}/requirements.yml \
+RUN pip3 install --no-cache-dir kubernetes~=25.3.0 \
+ && ansible-galaxy collection install -r ${HOME}/requirements.yml \
  && chmod -R ug+rwx ${HOME}/.ansible

 COPY watches.yaml ${HOME}/watches.yaml

$ sudo fips-mode-setup --check
FIPS mode is enabled.

$ kubectl -n awx logs deployments/awx-operator-controller-manager
...
PLAY RECAP *********************************************************************
localhost                  : ok=77   changed=0    unreachable=0    failed=0    skipped=74   rescued=0    ignored=1
...

jcmadden commented 1 year ago

Hi, I was able to upgrade to awx operator 1.2.0 on RHEL8.5 by implementing the solutions above:

going into the awx operator container after starting the upgrade
updating /opt/ansible/roles/common/tasks/main.yml to the fqcn for k8s
running pip3 install --no-cache-dir kubernetes~=25.3.0

However, I thought the issues may be resolved in awx operator 1.3.0 so I tried to upgrade to 1.3.0. I ran into a similar FIPS error for 1.3.0.

/opt/ansible/roles/installer/tasks/resources_configuration.yml:245\nok: [localhost] => {\"ansible_facts\": {\"_redis_image\": \"docker.io/redis:7\"}, \"changed\": false}\n\r\nTASK [installer : Apply deployment resources] **\r\ntask path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:249\nfatal: [localhost]: FAILED! => {\"msg\": \"An unhandled exception occurred while running the lookup plugin 'template'. Error was a <class 'ValueError'>, original message: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS\"}

I tried updating resources_configuration.yml to use the fqcn for template but it didn't work.

Please let me know if you have any suggestions.

Thanks You, Jack Madden

kurokobo commented 1 year ago

@jcmadden Another commit merged in 1.3.0 (https://github.com/ansible/awx-operator/commit/94d68bf382ec8dc4ce28d7d8d154663afd00b7fe) is causing your issue. This is already addressed and fixed in https://github.com/ansible/awx-operator/pull/1260, so it will work in the next release (SHA1 may not be sufficient, but it should work anyway).

jcmadden commented 1 year ago

@kurokobo Thank you very much. Implementing the solution from #1260 worked.

ansible / awx-operator