Latest image broken

[Mocem1]

Hi All,

The current latest image is not able to install from Ansible Galaxy.

The problem seems to be with Python cryptography 38. After manually downgrading to 37.0.4 it is working again.

Error message with Cryptography 38 `bash-4.4$ ansible-galaxy collection install networktocode.nautobot ERROR! Unexpected Exception, this is probably a bug: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK' the full traceback was:

Traceback (most recent call last): File "/usr/local/bin/ansible-galaxy", line 97, in mycli = getattr(import("ansible.cli.%s" % sub, fromlist=[myclass]), myclass) File "/usr/local/lib/python3.8/site-packages/ansible/cli/galaxy.py", line 24, in from ansible.galaxy.api import GalaxyAPI File "/usr/local/lib/python3.8/site-packages/ansible/galaxy/api.py", line 28, in from ansible.module_utils.urls import open_url, prepare_multipart File "/usr/local/lib/python3.8/site-packages/ansible/module_utils/urls.py", line 115, in from urllib3.contrib.pyopenssl import PyOpenSSLContext File "/usr/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 46, in import OpenSSL.SSL File "/usr/local/lib/python3.8/site-packages/OpenSSL/init.py", line 8, in from OpenSSL import crypto, SSL File "/usr/local/lib/python3.8/site-packages/OpenSSL/crypto.py", line 1517, in class X509StoreFlags(object): File "/usr/local/lib/python3.8/site-packages/OpenSSL/crypto.py", line 1537, in X509StoreFlags CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'`

Regards, M

[simonpahl]

Is it maybe possible to offer more tags to have a fallback to a previous working version in general. Or even a stable tag?

[baskraai]

+1 can confirm

[pkrul]

+1 for a stable tag.

Same issue here:

TASK [fetch galaxy roles from requirements.(yml/yaml)] ************************* failed: [localhost] (item=/var/lib/awx/projects/REDACTED/roles/requirements.yml) => {"ansible_loop_var": "item", "changed": false, "cmd": ["ansible-galaxy", "role", "install", "-r", "/var/lib/awx/projects/REDACTED/roles/requirements.yml", "--roles-path", "/var/lib/awx/projects/.__awx_cache/REDACTED/stage/requirements_roles"], "delta": "0:00:00.882823", "end": "2022-09-07 09:58:11.962114", "item": "/var/lib/awx/projects/REDACTED/roles/requirements.yml", "msg": "non-zero return code", "rc": 250, "start": "2022-09-07 09:58:11.079291", "stderr": "ERROR! Unexpected Exception, this is probably a bug: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'", "stderr_lines": ["ERROR! Unexpected Exception, this is probably a bug: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'"], "stdout": "the full traceback was:\n\nTraceback (most recent call last):\n File \"/usr/local/bin/ansible-galaxy\", li…

IMHO would be nice if we didn't have to create custom images to prevent this sort of breakage, etc., but the fast response is really appreciated :-)

[shrutebattlestargalactica]

i can confirm/report the same issues as the previous engineers - we created a separated awx-ee:0.6.0 image for our job templates and inventory source plugins to use this EE as a workaround

[baskraai]

@shrutebattlestargalactica you by any change able to share how you've done that for project jobs or is that out-of-scope? Been working on that all day but without any luck 🙁

[shrutebattlestargalactica]

from the inventory source we source from a project - just changed it to the custom EE i created - here is the EE I created with the "always" for the pull policy included

then from the inventory source - we source from a project (not sure if this is the same for you) and we use that EE for the inventory source

[baskraai]

@shrutebattlestargalactica thank you so much. Will try that on our instance!

[shrutebattlestargalactica]

@shrutebattlestargalactica thank you so much. Will try that on our instance!

Hope it helps - you should post if this is working for you. @baskraai

[djp928]

This also seems to be causing an issue with all inventory pulls from AWS with the following error:

[WARNING]: * Failed to parse /runner/inventory/aws_ec2.yml with auto plugin: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'

[sbrile]

Latest image also breaking vmware inventory.

[WARNING]: * Failed to parse /runner/project/dev/dev_vmware_vm_inventory.yml with auto plugin: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'

[pbanga-lumen]

Tried Galaxy install through quay.io/ansible/awx-ee:0.6.0 execution environment. Doesn't work. If anyone has a workaround other than custom image, please do let me know.

[mhorvat17]

When rolling back to quay.io/ansible/awx-ee:0.6.0 we are getting the error This can be caused if the version of ansible-runner in your execution environment is out of date.

Is there any way around that?

[MarcusCaepio]

from the inventory source we source from a project - just changed it to the custom EE i created - here is the EE I created !

Same error here since yesterday. Creating a custom AWX-EE does not work.

[luhahn]

I had to change the Control Plane EE as well for project updates to work.

Adding the following line to the _build/requirements.txt worked for me.

cryptography == 37.0.4

Edit: I just pushed to docker, feel free to use: novumrgi/awx-ee:stable

If you use the AWX-Operator for deployment add the following to your values:

image_pull_policy: Always
control_plane_ee_image: novumrgi/awx-ee:stable

[leetecarey]

Same issue here, inventory sync now failing

[Uquout7o]

@luhahn Thanks so much - I'm using your EE for now.

[JLE-ATS]

I have the same issue with the ansible.builtin.uri module using secure url (https).

[MarcusCaepio]

I had to change the Control Plane EE as well for project updates to work.

Adding the following line to the _build/requirements.txt worked for me.

cryptography == 37.0.4

Edit: I just pushed to docker, feel free to use: novumrgi/awx-ee:stable

If you use the AWX-Operator for deployment add the following to your values:

image_pull_policy: Always
control_plane_ee_image: novumrgi/awx-ee:stable

This should be fixed in the official repo asap....

[PaulVerhoeven1]

What i already thought would going to happen. There is a longstanding issue on GitHub about a new versiontag. https://github.com/ansible/awx-ee/issues/117 also this one is related: https://github.com/ansible/awx-ee/issues/125

[relrod]

I believe #136 should fix this for now. It does not solve the larger problem of not having tags other than latest to rely on, but it should at least unblock this issue.

Let's continue the tags discussion in #125/#117. :slightly_smiling_face:

[shrutebattlestargalactica]

sorry I should have mentioned we are on awx version 20.1 - I agree with @MarcusCaepio about fixing the official repo. I am still seeing issues with this on a dev instance that is on 21.5 regardless of the custom EE using the 0.6.0 image

@luhahn can you share what version of awx ur running?

[luhahn]

@shrutebattlestargalactica

Currently running on 21.5 ... Like I said, you also need to update the Control Plane EE, since this seems to be used for project updates.

[luhahn]

instead of my harbor stuff you can use novumrgi/awx-ee:stable , or the official one once it has been updated

[shrutebattlestargalactica]

i believe that the latest image is working again since it was updated 30 minutes ago

[mlkiefer]

As suggested above, I'd strongly suggest you pin the versions of all your python modules so that the result of your build process gets more reproducible.

If you don't, things like this will happen often in the future: A library gets an update that is not compatible to your software, the automatic builder will create an image and push it. In the newer versions of AWX, the latest image is being used (the newest release (about a year old) is not compatible. People that install their AWX after that date will get the non-working image. Even worse: because the image pull policy in the default execution environment is set to only pull images that are not present, they will stick with the non-working version unless they change this setting. Others that were lucky enough to install their AWX at a time when a working image was present, will not notice this until maybe their AWX is not compatible with an EE image that is sticking around for too long because it is set to LATEST and not always pulling the newest version of a >500MB image.

=> using LATEST tags and not pinning versions in productive environments is evil.

Please consider releasing more tagged images, as the first responder to this issue sugested here: https://github.com/ansible/awx-ee/issues/134#issuecomment-1239150226

[pbanga-lumen]

I am not sure if this is fixed or I am not doing something right. I pulled the latest repo did make deploy and applied awx defaults. I still get same error when I try to install modules through ansible-galaxy.

git clone https://github.com/ansible/awx-operator.git

`awx-user@ubuntu-awx:~/awx-operator$ kubectl -n awx get awx -o yaml awx apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"awx.ansible.com/v1beta1","kind":"AWX","metadata":{"annotations":{},"name":"awx","namespace":"awx"},"spec":{"projects_persistence":true,"projects_storage_access_mode":"ReadWriteOnce","service_type":"nodeport"}} creationTimestamp: "2022-09-09T00:16:31Z" generation: 1 labels: app.kubernetes.io/component: awx app.kubernetes.io/managed-by: awx-operator app.kubernetes.io/name: awx app.kubernetes.io/operator-version: 0.28.0 app.kubernetes.io/part-of: awx name: awx namespace: awx resourceVersion: "108821" uid: df19f1bf-5e60-45ee-96e5-825e4d60cabc spec: admin_user: admin auto_upgrade: true create_preload_data: true garbage_collect_secrets: false image_pull_policy: IfNotPresent loadbalancer_port: 80 loadbalancer_protocol: http nodeport_port: 30080 projects_persistence: true projects_storage_access_mode: ReadWriteOnce projects_storage_size: 8Gi replicas: 1 route_tls_termination_mechanism: Edge service_type: nodeport set_self_labels: true task_privileged: false status: adminPasswordSecret: awx-admin-password adminUser: admin broadcastWebsocketSecret: awx-broadcast-websocket conditions:

• lastTransitionTime: "2022-09-09T00:20:43Z" reason: "" status: "False" type: Failure
• lastTransitionTime: "2022-09-09T00:16:31Z" reason: Successful status: "True" type: Running
• lastTransitionTime: "2022-09-09T00:21:47Z" reason: Successful status: "True" type: Successful image: quay.io/ansible/awx:21.5.0 postgresConfigurationSecret: awx-postgres-configuration secretKeySecret: awx-secret-key version: 21.5.0 awx-user@ubuntu-awx:~/awx-operator$`

Job result `/usr/local/lib/python3.8/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated "class": algorithms.Blowfish,

PLAY [Update source tree if necessary] *****

TASK [update project using git] **** ok: [localhost]

TASK [Set the git repository version] ** ok: [localhost]

TASK [Repository Version] ** ok: [localhost] => { "msg": "Repository Version f7feffab95d2a7b0986d9eb03057cf69db74986a" }

PLAY [Install content with ansible-galaxy command if necessary] ****

TASK [debug] *** skipping: [localhost]

TASK [meta] **** skipping: [localhost]

TASK [fetch galaxy roles from requirements.(yml/yaml)] ***** [WARNING]: Unable to find '/var/lib/awx/projects/_8__xxxxx_demo_project/roles' in expected paths (use -vvvvv to see paths)

TASK [fetch galaxy collections from collections/requirements.(yml/yaml)] *** failed: [localhost] (item=/var/lib/awx/projects/_8xxxxx_demo_project/collections/requirements.yml) => {"ansible_loop_var": "item", "changed": false, "cmd": ["ansible-galaxy", "collection", "install", "-r", "/var/lib/awx/projects/_8xxxxx_demo_project/collections/requirements.yml", "--collections-path", "/var/lib/awx/projects/.awx_cache/_8xxxxx_demo_project/stage/requirements_collections"], "delta": "0:00:00.615822", "end": "2022-09-09 00:22:39.327397", "item": "/var/lib/awx/projects/_8xxxxx_demo_project/collections/requirements.yml", "msg": "non-zero return code", "rc": 250, "start": "2022-09-09 00:22:38.711575", "stderr": "ERROR! Unexpected Exception, this is probably a bug: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'", "stderr_lines": ["ERROR! Unexpected Exception, this is probably a bug: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'"], "stdout": "the full traceback was:\n\nTraceback (most recent call last):\n File \"/usr/local/bin/ansible-galaxy\", line 97, in \n mycli = getattr(import(\"ansible.cli.%s\" % sub, fromlist=[myclass]), myclass)\n File \"/usr/local/lib/python3.8/site-packages/ansible/cli/galaxy.py\", line 24, in \n from ansible.galaxy.api import GalaxyAPI\n File \"/usr/local/lib/python3.8/site-packages/ansible/galaxy/api.py\", line 28, in \n from ansible.module_utils.urls import open_url, prepare_multipart\n File \"/usr/local/lib/python3.8/site-packages/ansible/module_utils/urls.py\", line 115, in \n from urllib3.contrib.pyopenssl import PyOpenSSLContext\n File \"/usr/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py\", line 46, in \n import OpenSSL.SSL\n File \"/usr/local/lib/python3.8/site-packages/OpenSSL/init.py\", line 8, in \n from OpenSSL import crypto, SSL\n File \"/usr/local/lib/python3.8/site-packages/OpenSSL/crypto.py\", line 1517, in \n class X509StoreFlags(object):\n File \"/usr/local/lib/python3.8/site-packages/OpenSSL/crypto.py\", line 1537, in X509StoreFlags\n CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK\nAttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'", "stdout_lines": ["the full traceback was:", "", "Traceback (most recent call last):", " File \"/usr/local/bin/ansible-galaxy\", line 97, in ", " mycli = getattr(import(\"ansible.cli.%s\" % sub, fromlist=[myclass]), myclass)", " File \"/usr/local/lib/python3.8/site-packages/ansible/cli/galaxy.py\", line 24, in ", " from ansible.galaxy.api import GalaxyAPI", " File \"/usr/local/lib/python3.8/site-packages/ansible/galaxy/api.py\", line 28, in ", " from ansible.module_utils.urls import open_url, prepare_multipart", " File \"/usr/local/lib/python3.8/site-packages/ansible/module_utils/urls.py\", line 115, in ", " from urllib3.contrib.pyopenssl import PyOpenSSLContext", " File \"/usr/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py\", line 46, in ", " import OpenSSL.SSL", " File \"/usr/local/lib/python3.8/site-packages/OpenSSL/init__.py\", line 8, in ", " from OpenSSL import crypto, SSL", " File \"/usr/local/lib/python3.8/site-packages/OpenSSL/crypto.py\", line 1517, in ", " class X509StoreFlags(object):", " File \"/usr/local/lib/python3.8/site-packages/OpenSSL/crypto.py\", line 1537, in X509StoreFlags", " CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK", "AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'"]}

PLAY RECAP ***** localhost : ok=3 changed=0 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0 `

[shrutebattlestargalactica]

@luhahn Hey there - just curious if you are seeing any problems with the latest quay image again

[luhahn]

@shrutebattlestargalactica to be honest we haven't bothered updating the image for now, since we've moved to a less AWX based approach for ansible. We're still using the stable image I've published back in 2022.