[TRACKER] 9.1.0 collection test failures

[gotmax23]

(All the collections that have been retired from Ansible 10.3.0 or otherwise no longer have sanity test failures in that version should be checked off.)

./main.py -i data/9/ansible-9.1.0-sanity.yaml -t build/ansible-build-data/9/ansible-9.1.0-tags.yaml -u data/9/ansible-9.1.0-missing-upstreams.yaml failures --bullet="- [ ]" --reverse | wl-copy

• [ ] cisco.intersight -- https://github.com/CiscoDevNet/intersight-ansible/issues/108 15512 validate-modules
• [x] fortinet.fortimanager -- https://github.com/fortinet-ansible-dev/ansible-galaxy-fortimanager-collection/issues/80 13015 validate-modules
• [ ] cisco.ucs -- https://github.com/CiscoDevNet/ansible-ucs/issues/40 9502 validate-modules
• [x] ovirt.ovirt (upstream runs the tests but has their own script to preprocess their sources which we cannot run here) 136 WRONG_HASH 113 yamllint 66 validate-modules 60 compile-python-3.10 60 compile-python-3.11 60 compile-python-3.12 59 compile-python-2.7 59 compile-python-3.6 59 compile-python-3.7 59 compile-python-3.8 59 compile-python-3.9 11 ansible-doc 1 WRONG_GALAXY_YML_NAME
• [x] community.network -- https://github.com/ansible-collections/community.network/issues/591 (exclusion discussion started in https://forum.ansible.com/t/should-we-remove-community-network-from-the-ansible-community-package/8030) 654 BANNED_SANITY
• [ ] check_point.mgmt -- https://github.com/CheckPointSW/CheckPointAnsibleMgmtCollection/issues/121, https://github.com/CheckPointSW/CheckPointAnsibleMgmtCollection/issues/142 240 validate-modules 120 WRONG_HASH 3 ansible-doc
• [ ] netapp_eseries.santricity -- https://github.com/NetApp/santricity/issues/23 342 validate-modules 3 ansible-doc 2 yamllint
• [x] azure.azcollection -- https://github.com/ansible-collections/azure/issues/1417 311 validate-modules
• [ ] google.cloud -- https://github.com/ansible-collections/google.cloud/issues/613 186 validate-modules
• [x] theforeman.foreman -- https://github.com/theforeman/foreman-ansible-modules/issues/1703 (upstream runs the tests but has their own script to preprocess their sources which we cannot run here) 153 yamllint
• [ ] wti.remote -- https://github.com/wtinetworkgear/wti-collection/issues/15 85 validate-modules 7 ansible-doc 4 yamllint
• [x] hpe.nimble -- https://github.com/hpe-storage/nimble-ansible-modules/issues/63 25 validate-modules 23 compile-python-2.7
• [x] infinidat.infinibox -- https://github.com/Infinidat/ansible-infinidat-collection/issues/19 24 validate-modules 3 compile-python-3.10 3 compile-python-3.11 3 compile-python-3.12 1 compile-python-2.7
• [ ] kubernetes.core -- https://github.com/ansible-collections/kubernetes.core/issues/672 25 validate-modules 9 yamllint
• [x] cisco.ise -- https://github.com/CiscoISE/ansible-ise/issues/117 15 validate-modules 7 ansible-doc 1 yamllint
• [x] dellemc.unity -- https://github.com/dell/ansible-unity/issues/47 18 validate-modules 3 compile-python-2.7
• [x] netapp.azure -- https://github.com/ansible-collections/netapp.azure/issues/9 (removed) 16 validate-modules
• [x] gluster.gluster -- https://github.com/gluster/gluster-ansible-collection/issues/31 10 validate-modules
• [x] community.sap_libs -- https://github.com/sap-linuxlab/community.sap_libs/issues/38 9 validate-modules
• [ ] sensu.sensu_go -- https://github.com/sensu/sensu-go-ansible/issues/362 7 validate-modules 1 WRONG_HASH
• [x] community.okd -- https://github.com/openshift/community.okd/issues/219 7 validate-modules
• [ ] openstack.cloud -- https://bugs.launchpad.net/ansible-collections-openstack/+bug/2050854 5 compile-python-2.7 2 yamllint
• [x] community.digitalocean -- https://github.com/ansible-collections/community.digitalocean/issues/347 4 validate-modules 2 compile-python-2.7
• [x] purestorage.fusion -- https://github.com/Pure-Storage-Ansible/Fusion-Collection/issues/166 6 compile-python-2.7
• [x] netapp.um_info -- https://github.com/ansible-collections/netapp.um_info/issues/11 (removed) 5 validate-modules
• [x] community.general -- https://github.com/ansible-collections/community.general/issues/7884 4 BANNED_SANITY
• [x] community.mysql -- https://github.com/ansible-collections/community.mysql/issues/608 4 BANNED_SANITY
• [x] netapp.aws -- https://github.com/ansible-collections/netapp.aws/issues/9 (removed) 4 validate-modules
• [x] ansible.posix -- https://github.com/ansible-collections/ansible.posix/issues/528 3 BANNED_SANITY
• [x] chocolatey.chocolatey -- https://github.com/chocolatey/chocolatey-ansible/issues/144 3 validate-modules
• [ ] cyberark.conjur -- https://github.com/cyberark/ansible-conjur-collection/issues/201 2 yamllint 1 validate-modules
• [x] ibm.spectrum_virtualize (DEPRECATED) 3 WRONG_HASH
• [ ] awx.awx 1 BANNED_SANITY 1 WRONG_HASH
• [x] cisco.mso 2 validate-modules
• [ ] community.mongodb 2 yamllint
• [x] community.sap -- https://github.com/ansible-collections/community.sap/issues/32 2 yamllint
• [x] lowlydba.sqlserver 2 WRONG_HASH
• [ ] netbox.netbox 2 compile-python-2.7
• [x] vmware.vmware_rest 2 validate-modules
• [x] cisco.asa 1 compile-python-2.7
• [x] cisco.dnac 1 compile-python-2.7
• [x] community.postgresql -- Fixed https://github.com/ansible-collections/community.postgresql/pull/664 1 BANNED_SANITY
• [x] community.zabbix 1 compile-python-2.7
• [x] containers.podman 1 compile-python-2.7
• [x] netapp.storagegrid 1 validate-modules
• [x] purestorage.flasharray 1 compile-python-2.7

[mariolenz]

I think you can ignore some of those collections since they'll be removed from Ansible 10 anyway:

• ansible-community/ansible-build-data#243
• ansible-community/ansible-build-data#249
• ansible-community/ansible-build-data#266
• ansible-community/ansible-build-data#272
• ansible-community/ansible-build-data#280
• ansible-community/ansible-build-data#285

[mariolenz]

BTW I see purestorage.fusion on your list. It looks like this collection will be deprecated. So maybe you can ignore this, too.

[mariolenz]

BTW I see purestorage.fusion on your list. It looks like this collection will be deprecated. So maybe you can ignore this, too.

@gotmax23 This collection has been deprecated by it's maintainers and will be removed from Ansible 10: ansible-community/ansible-build-data#374

[gotmax23]

Results for 10.3.0 are listed below and also represented in https://github.com/ansible-community/package-test-results/tree/main/rendered/10.3.0. Some of the duplicates from above may have been fixed in git but never released. Other have not fixed all the issues or not responded at all.

(Duplicates that are already listed in the issue description and that have had issues filled already should be checked off here.)

• [x] cisco.intersight 15582 validate-modules
• [x] cisco.ucs 9502 validate-modules
• [x] ovirt.ovirt 136 WRONG_HASH 113 yamllint 66 validate-modules 60 compile-python-3.10 60 compile-python-3.11 60 compile-python-3.12 59 compile-python-3.7 59 compile-python-3.8 59 compile-python-3.9 11 ansible-doc 1 WRONG_GALAXY_YML_NAME
• [x] community.network 654 BANNED_SANITY
• [x] netapp_eseries.santricity 342 validate-modules 3 ansible-doc 2 yamllint
• [x] check_point.mgmt 261 validate-modules 4 WRONG_HASH
• [x] google.cloud 186 validate-modules
• [x] theforeman.foreman 122 yamllint
• [x] wti.remote 85 validate-modules 7 ansible-doc 4 yamllint
• [ ] ansible.posix 9 validate-modules
• [x] sensu.sensu_go 7 validate-modules 1 WRONG_HASH
• [ ] netapp.ontap 4 validate-modules
• [x] chocolatey.chocolatey 3 validate-modules
• [x] cyberark.conjur 2 yamllint 1 validate-modules
• [ ] cyberark.pas 2 WRONG_HASH 1 validate-modules
• [ ] ibm.spectrum_virtualize 3 WRONG_HASH
• [x] kubernetes.core 3 BANNED_SANITY
• [ ] awx.awx 1 BANNED_SANITY 1 WRONG_HASH
• [ ] community.mongodb 2 yamllint
• [ ] community.okd 2 BANNED_SANITY
• [ ] ibm.storage_virtualize 2 WRONG_HASH
• [x] inspur.ispim 2 WRONG_HASH
• [ ] kubevirt.core 2 WRONG_HASH
• [ ] openstack.cloud 2 yamllint

[gotmax23]

Followed up with:

• [ ] https://github.com/wtinetworkgear/wti-collection/issues/15
• [ ] https://github.com/cyberark/ansible-conjur-collection/issues/201
• [ ] https://github.com/sensu/sensu-go-ansible/issues/362
• [ ] https://github.com/ansible-collections/google.cloud/issues/613

(Will follow up again in a couple weeks)

[gotmax23]

@gundalow followed up with

• [ ] https://github.com/NetApp/santricity/issues/23

(We should follow up again in a couple weeks)

[gotmax23]

• [ ] https://github.com/CiscoDevNet/ansible-ucs/issues/40 (current results at https://github.com/ansible-community/package-test-results/blob/main/rendered/10.3.0/cisco.ucs.md)
• [ ] https://github.com/CiscoDevNet/intersight-ansible/issues/108 (current results at https://github.com/ansible-community/package-test-results/blob/main/rendered/10.3.0/cisco.intersight.md)

are also a problem. The maintainer says that our results are broken and differ from the results reported by Automation Hub. Perhaps there are sanity ignore entries that are not included in the git repository. Otherwise, I am at a loss.

[gotmax23]

@gundalow, any idea what's going on with the Cisco collections? Looking at the output in https://console.redhat.com/ansible/automation-hub/repo/published/cisco/intersight/import-log/, it seems that AH doesn't run the validate-modules test which is one that we require to pass.

[gotmax23]

A discussion about community.network has been started in https://forum.ansible.com/t/should-we-remove-community-network-from-the-ansible-community-package/8030.

[felixfontein]

• [ ] Community package requirements: sanity tests CiscoDevNet/ansible-ucs#40 (current results at https://github.com/ansible-community/package-test-results/blob/main/rendered/10.3.0/cisco.ucs.md)

  • [ ] Community package requirements: sanity tests CiscoDevNet/intersight-ansible#108 (current results at https://github.com/ansible-community/package-test-results/blob/main/rendered/10.3.0/cisco.intersight.md)

are also a problem. The maintainer says that our results are broken and differ from the results reported by Automation Hub. Perhaps there are sanity ignore entries that are not included in the git repository. Otherwise, I am at a loss.

It turned out that this is actually due to a bug in ansible-test (https://github.com/ansible/ansible/issues/83842). On the other hand, if Automation Hub's sanity tests accepted it, it means that AH doesn't run the validate-modules sanity test. This basically means that successful import in AH doesn't mean that all sanity tests pass. (But it also doesn't mean that the ones not run fail...)