Failure building Ansible Execution Environment due to incompatible python modules from azure.azcollection

smeyer01 commented 1 year ago

SUMMARY

Attempting to build an Ansible Execution Environment using Ansible Builder with verbose output (-v 3) results in the following message during the build:

pyopenssl 22.0.0 requires cryptography>=35.0, but you'll have cryptography 2.8 which is incompatible.
azure-cli-telemetry 1.0.6 requires portalocker~=1.2, but you'll have portalocker 2.5.1 which is incompatible.
azure-mgmt-datalake-nspkg 3.0.1 requires azure-mgmt-nspkg>=3.0.0, but you'll have azure-mgmt-nspkg 2.0.0 which is incompatible.

This does not produce a fatal error in the build, so the build continues and looks to be clean. The build completes, however the messages indicate an issue which prevent the azure collection from working properly.

As a temporary workaround I was able to include these in a new file requirements-python.txt which built without the incompatible python module errors:

cryptography==37.0.4
azure-mgmt-datalake-nspkg==2.0.0
portalocker==1.7.1

ISSUE TYPE

Bug Report

COMPONENT NAME

azure.azcollection module including azure.azcollection.azure_rm_resourcegroup

ANSIBLE VERSION

ansible-builder --version
1.0.1

COLLECTION VERSION

1.12.0

CONFIGURATION

n/a

OS / ENVIRONMENT

Red Hat Enterprise Linux release 8.6 (Ootpa) container image

STEPS TO REPRODUCE

ansible-builder build --file execution-environment.yml -t bscee:0.0.12 -v 3

requirements-collections.yml:

---
collections:
- name: azure.azcollection
  version: 1.12.0
  source: https://cloud.redhat.com/api/automation-hub/

execution-enviroment.yml

version: 1

build_arg_defaults:
  EE_BASE_IMAGE: 'registry.redhat.io/ansible-automation-platform-21/ee-minimal-rhel8:latest'

ansible_config: 'ansible.cfg'

dependencies:
  galaxy: requirements-collections.yml
  system: bindep.txt

bindep.txt

dnf [platform:rpm]
git [platform:rpm]
jq [platform:rpm]

EXPECTED RESULTS

The ansible-builder should complete with having to make adjustments in requirements-python.txt. It should build cleanly without any incompatible module messages with its own requirements.txt file provided within the module from Microsoft.

ACTUAL RESULTS

During the build using ansible-builder, the output contains incompatible python module messages, which hinder the usability of the resulting container image.

ansiblejunky commented 1 year ago

I ran ansible-builder today to verify this issue is happening with release 1.12.0 of the azure collection. Here is the errors I'm seeing ... again, we are building an Execution Environment using the registry.redhat.io/ansible-automation-platform-22/ee-minimal-rhel8:latest base image.

#22 22.69 Requirement already satisfied: ply==3.11 in /usr/lib/python3.9/site-packages (from pycparser->cffi>=1.12->cryptography->azure-cli-core==2.34.0->-r /output/requirements.txt (line 4)) (3.11)
#22 22.69 Using legacy 'setup.py install' for pyperclip, since package 'wheel' is not installed.
#22 23.21 Installing collected packages: PyJWT, msal, pkginfo, portalocker, applicationinsights, azure-cli-telemetry, psutil, msal-extensions, jmespath, humanfriendly, tabulate, argcomplete, knack, typing-extensions, azure-core, azure-mgmt-core, python-dateutil, adal, certifi, oauthlib, requests-oauthlib, isodate, msrest, msrestazure, azure-cli-core, azure-nspkg, azure-common, azure-identity, azure-mgmt-apimanagement, azure-mgmt-authorization, azure-mgmt-nspkg, azure-mgmt-batch, azure-mgmt-cdn, azure-mgmt-compute, azure-mgmt-containerinstance, azure-mgmt-containerregistry, azure-mgmt-containerservice, azure-mgmt-datalake-nspkg, azure-mgmt-datalake-store, azure-mgmt-dns, azure-mgmt-keyvault, azure-mgmt-marketplaceordering, azure-mgmt-monitor, azure-mgmt-managedservices, azure-mgmt-managementgroups, azure-mgmt-network, azure-mgmt-privatedns, azure-mgmt-redis, azure-mgmt-resource, azure-mgmt-rdbms, azure-mgmt-search, azure-mgmt-servicebus, azure-mgmt-sql, azure-mgmt-storage, azure-mgmt-trafficmanager, azure-mgmt-web, azure-storage, azure-keyvault, azure-graphrbac, azure-mgmt-cosmosdb, azure-mgmt-hdinsight, azure-mgmt-devtestlabs, azure-mgmt-loganalytics, azure-mgmt-automation, azure-mgmt-iothub, azure-mgmt-recoveryservices, azure-mgmt-recoveryservicesbackup, azure-mgmt-notificationhubs, azure-mgmt-eventhub, pbr, stevedore, attrs, wcwidth, pyperclip, cmd2, PrettyTable, zipp, importlib-metadata, autopage, cliff, ara
#22 53.65     Running setup.py install for pyperclip: started
#22 53.90     Running setup.py install for pyperclip: finished with status 'done'
#22 54.25 Successfully installed PrettyTable-3.4.1 PyJWT-2.5.0 adal-1.2.7 applicationinsights-0.11.10 ara-1.5.8 argcomplete-1.12.3 attrs-22.1.0 autopage-0.5.1 azure-cli-core-2.34.0 azure-cli-telemetry-1.0.6 azure-common-1.1.11 azure-core-1.25.1 azure-graphrbac-0.61.1 azure-identity-1.7.0 azure-keyvault-1.0.0a1 azure-mgmt-apimanagement-0.2.0 azure-mgmt-authorization-0.51.1 azure-mgmt-automation-0.1.1 azure-mgmt-batch-5.0.1 azure-mgmt-cdn-3.0.0 azure-mgmt-compute-23.1.0 azure-mgmt-containerinstance-1.4.0 azure-mgmt-containerregistry-2.0.0 azure-mgmt-containerservice-9.1.0 azure-mgmt-core-1.3.2 azure-mgmt-cosmosdb-0.5.2 azure-mgmt-datalake-nspkg-3.0.1 azure-mgmt-datalake-store-0.5.0 azure-mgmt-devtestlabs-3.0.0 azure-mgmt-dns-2.1.0 azure-mgmt-eventhub-2.0.0 azure-mgmt-hdinsight-0.1.0 azure-mgmt-iothub-0.7.0 azure-mgmt-keyvault-1.1.0 azure-mgmt-loganalytics-1.0.0 azure-mgmt-managedservices-1.0.0 azure-mgmt-managementgroups-0.2.0 azure-mgmt-marketplaceordering-0.1.0 azure-mgmt-monitor-3.0.0 azure-mgmt-network-19.1.0 azure-mgmt-notificationhubs-2.0.0 azure-mgmt-nspkg-2.0.0 azure-mgmt-privatedns-0.1.0 azure-mgmt-rdbms-1.9.0 azure-mgmt-recoveryservices-0.4.0 azure-mgmt-recoveryservicesbackup-0.6.0 azure-mgmt-redis-5.0.0 azure-mgmt-resource-10.2.0 azure-mgmt-search-3.0.0 azure-mgmt-servicebus-0.5.3 azure-mgmt-sql-0.10.0 azure-mgmt-storage-19.0.0 azure-mgmt-trafficmanager-0.50.0 azure-mgmt-web-0.41.0 azure-nspkg-2.0.0 azure-storage-0.35.1 certifi-2022.9.24 cliff-4.0.0 cmd2-2.4.2 humanfriendly-10.0 importlib-metadata-4.12.0 isodate-0.6.1 jmespath-1.0.1 knack-0.9.0 msal-1.19.0 msal-extensions-0.3.1 msrest-0.6.21 msrestazure-0.6.4 oauthlib-3.2.1 pbr-5.10.0 pkginfo-1.8.3 portalocker-1.7.1 psutil-5.9.2 pyperclip-1.8.2 python-dateutil-2.8.2 requests-oauthlib-1.3.1 stevedore-4.0.0 tabulate-0.8.10 typing-extensions-4.3.0 wcwidth-0.2.5 zipp-3.8.1
#22 54.25 ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.
#22 54.25 
#22 54.25 We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.
#22 54.25 
#22 54.25 azure-mgmt-datalake-nspkg 3.0.1 requires azure-mgmt-nspkg>=3.0.0, but you'll have azure-mgmt-nspkg 2.0.0 which is incompatible.
#22 54.40 + EXTRAS=
#22 54.40 + '[' -f /output/packages.txt ']'
#22 54.40 ++ ls -1 '/output/wheels/*whl'
#22 54.40 ++ wc -l
#22 54.40 + '[' 0 -gt 0 ']'
#22 54.40 + '[' '!' -z '' ']'
#22 54.40 + /usr/bin/microdnf clean all
#22 54.43 
#22 54.43 (microdnf:139): librhsm-WARNING **: 22:02:13.107: Found 0 entitlement certificates
#22 54.43 
#22 54.43 (microdnf:139): librhsm-WARNING **: 22:02:13.108: Found 0 entitlement certificates
#22 54.44 Complete.
#22 54.45 + rm -rf /var/cache/dnf /var/cache/yum
#22 54.45 + rm -rf /var/lib/dnf/history.sqlite /var/lib/dnf/history.sqlite-shm /var/lib/dnf/history.sqlite-wal
#22 54.45 + rm -rf '/var/log/dnf.*' /var/log/hawkey.log
#22 DONE 55.5s

ansiblejunky commented 1 year ago

Note that there are no incompatible errors when referencing version 1.11.0 from the collection.

p3ck commented 5 months ago

Have you tried this again with AAP-2.4 and azure-azcollection-2.2.0 ?

There still is an issue with azure_mgmt_datalake_store-1.0.0 which has been referenced here: https://github.com/ansible-collections/azure/issues/1072

But other than that it builds fine.

ansible-collections / azure