Deprecation warnings when building ansible with setuptools-62.6.0

[gotmax23]

When running setup,py build while building ansible against Fedora Rawhide (which has setuptools-62.6.0), the following error is printed for every single collection (and their sub-directories):

/usr/lib/python3.11/site-packages/setuptools/command/build_py.py:153: SetuptoolsDeprecationWarning:     Installing 'ansible_collections.amazon.aws' as data is deprecated, please list it in `packages`.

    !!
    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'ansible_collections.amazon.aws' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.
    'ansible_collections.amazon.aws' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).
    Please make sure that 'ansible_collections.amazon.aws' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).
    You can read more about "package discovery" and "data files" on setuptools
    documentation page.
!!

I'm not going to include the whole output, as there are 185867 lines worth of these warnings. See here for all the deprecation warnings and here for the full rpmbuild log.

[felixfontein]

Hmm, but actually we want to install these files as data, and not as Python packages.

If I understand it correctly, we will have to list

    packages=[
        'ansible_collections',
        'ansible_collections.amazon',
        'ansible_collections.amazon.aws',
        'ansible_collections.amazon.aws.changelog',
        'ansible_collections.amazon.aws.docs',
        ...
    ],

(i.e. list every single directory) instead of just

    packages=['ansible_collections'],

as until now.

I don't think we can avoid listing all directories by using find_packages() since almost all directories will not have an __init__.py file in them.

What might work is using find_namespace_packages() though, since at least its implementation seems to be doing exactly what we need (list all directories, not just the ones having an __init__.py in them).

(Also we need a mechanism to show these deprecation warnings. Right now they are hidden when antsibull calls setup.py, as it simply throws away all output.)

[felixfontein]

(I moved this into the antsibull repository since this is something that can and must be fixed by antsibull.)

[felixfontein]

This will be fixed by #434.

[gotmax23]

I think I've finally figured this out.

@@ -5,7 +5,7 @@

 import os
 import sys
-from setuptools import setup
+from setuptools import setup, find_namespace_packages

 __version__ = '7.2.0'
 __author__ = 'Ansible, Inc.'
@@ -31,260 +31,18 @@
     },
     license='GPLv3+',
     python_requires='>=3.9',
-    packages=['ansible_collections'],
-
-    exclude_package_data={
-        'ansible_collections': [
-            'amazon/aws/.*',
-            'amazon/aws/docs/*',
-            'amazon/aws/tests/*',
-            'ansible/netcommon/.*',
-            'ansible/netcommon/docs/*',
-            'ansible/netcommon/tests/*',
-            'ansible/posix/.*',
-            'ansible/posix/docs/*',
-            'ansible/posix/tests/*',
-            'ansible/utils/.*',
-            'ansible/utils/docs/*',
-            'ansible/utils/tests/*',
-            'ansible/windows/.*',
-            'ansible/windows/docs/*',
-            'ansible/windows/tests/*',
-            'arista/eos/.*',
-            'arista/eos/docs/*',
-            'arista/eos/tests/*',
-            'awx/awx/tests/*',
-            'azure/azcollection/.*',
-            'azure/azcollection/tests/*',
-            'check_point/mgmt/tests/*',
-            'chocolatey/chocolatey/tests/*',
-            'cisco/aci/.*',
-            'cisco/aci/tests/*',
-            'cisco/asa/.*',
-            'cisco/asa/docs/*',
-            'cisco/asa/tests/*',
-            'cisco/dnac/.*',
-            'cisco/dnac/docs/*',
-            'cisco/dnac/tests/*',
-            'cisco/intersight/.*',
-            'cisco/ios/.*',
-            'cisco/ios/docs/*',
-            'cisco/ios/tests/*',
-            'cisco/iosxr/.*',
-            'cisco/iosxr/docs/*',
-            'cisco/iosxr/tests/*',
-            'cisco/ise/.*',
-            'cisco/ise/docs/*',
-            'cisco/ise/tests/*',
-            'cisco/meraki/.*',
-            'cisco/meraki/tests/*',
-            'cisco/mso/.*',
-            'cisco/mso/tests/*',
-            'cisco/nso/.*',
-            'cisco/nso/tests/*',
-            'cisco/nxos/.*',
-            'cisco/nxos/docs/*',
-            'cisco/nxos/tests/*',
-            'cisco/ucs/.*',
-            'cloud/common/tests/*',
-            'cloudscale_ch/cloud/.*',
-            'cloudscale_ch/cloud/tests/*',
-            'community/aws/.*',
-            'community/aws/docs/*',
-            'community/aws/tests/*',
-            'community/azure/.*',
-            'community/azure/tests/*',
-            'community/ciscosmb/.*',
-            'community/ciscosmb/tests/*',
-            'community/crypto/.*',
-            'community/crypto/docs/*',
-            'community/crypto/tests/*',
-            'community/digitalocean/.*',
-            'community/digitalocean/tests/*',
-            'community/dns/.*',
-            'community/dns/docs/*',
-            'community/dns/tests/*',
-            'community/docker/.*',
-            'community/docker/docs/*',
-            'community/docker/tests/*',
-            'community/fortios/.*',
-            'community/fortios/tests/*',
-            'community/general/.*',
-            'community/general/docs/*',
-            'community/general/tests/*',
-            'community/google/.*',
-            'community/google/tests/*',
-            'community/grafana/.*',
-            'community/grafana/tests/*',
-            'community/hashi_vault/.*',
-            'community/hashi_vault/docs/*',
-            'community/hashi_vault/tests/*',
-            'community/hrobot/.*',
-            'community/hrobot/docs/*',
-            'community/hrobot/tests/*',
-            'community/libvirt/.*',
-            'community/libvirt/tests/*',
-            'community/mongodb/.*',
-            'community/mysql/.*',
-            'community/mysql/tests/*',
-            'community/network/.*',
-            'community/network/tests/*',
-            'community/okd/.*',
-            'community/okd/docs/*',
-            'community/okd/tests/*',
-            'community/postgresql/.*',
-            'community/postgresql/docs/*',
-            'community/postgresql/tests/*',
-            'community/proxysql/.*',
-            'community/proxysql/tests/*',
-            'community/rabbitmq/.*',
-            'community/rabbitmq/tests/*',
-            'community/routeros/.*',
-            'community/routeros/docs/*',
-            'community/routeros/tests/*',
-            'community/sap/.*',
-            'community/sap/tests/*',
-            'community/sap_libs/.*',
-            'community/sap_libs/tests/*',
-            'community/skydive/.*',
-            'community/skydive/tests/*',
-            'community/sops/.*',
-            'community/sops/docs/*',
-            'community/sops/tests/*',
-            'community/vmware/.*',
-            'community/vmware/docs/*',
-            'community/vmware/tests/*',
-            'community/windows/.*',
-            'community/windows/docs/*',
-            'community/windows/tests/*',
-            'community/zabbix/.*',
-            'community/zabbix/docs/*',
-            'community/zabbix/tests/*',
-            'containers/podman/docs/*',
-            'containers/podman/tests/*',
-            'cyberark/conjur/.*',
-            'cyberark/conjur/tests/*',
-            'cyberark/pas/.*',
-            'cyberark/pas/docs/*',
-            'cyberark/pas/tests/*',
-            'dellemc/enterprise_sonic/.*',
-            'dellemc/enterprise_sonic/tests/*',
-            'dellemc/openmanage/.*',
-            'dellemc/openmanage/docs/*',
-            'dellemc/openmanage/tests/*',
-            'dellemc/os10/.*',
-            'dellemc/os10/docs/*',
-            'dellemc/os10/tests/*',
-            'dellemc/os6/.*',
-            'dellemc/os6/docs/*',
-            'dellemc/os6/tests/*',
-            'dellemc/os9/.*',
-            'dellemc/os9/docs/*',
-            'dellemc/os9/tests/*',
-            'dellemc/powerflex/docs/*',
-            'dellemc/powerflex/tests/*',
-            'dellemc/unity/docs/*',
-            'dellemc/unity/tests/*',
-            'f5networks/f5_modules/docs/*',
-            'f5networks/f5_modules/tests/*',
-            'fortinet/fortimanager/tests/*',
-            'frr/frr/.*',
-            'frr/frr/docs/*',
-            'frr/frr/tests/*',
-            'gluster/gluster/.*',
-            'gluster/gluster/tests/*',
-            'google/cloud/.*',
-            'google/cloud/tests/*',
-            'grafana/grafana/.*',
-            'grafana/grafana/tests/*',
-            'hetzner/hcloud/.*',
-            'hetzner/hcloud/tests/*',
-            'hpe/nimble/.*',
-            'hpe/nimble/tests/*',
-            'ibm/qradar/.*',
-            'ibm/qradar/docs/*',
-            'ibm/qradar/tests/*',
-            'ibm/spectrum_virtualize/.*',
-            'ibm/spectrum_virtualize/docs/*',
-            'ibm/spectrum_virtualize/tests/*',
-            'infinidat/infinibox/.*',
-            'infinidat/infinibox/docs/*',
-            'infinidat/infinibox/tests/*',
-            'infoblox/nios_modules/.*',
-            'infoblox/nios_modules/tests/*',
-            'inspur/ispim/.*',
-            'inspur/ispim/docs/*',
-            'inspur/sm/.*',
-            'inspur/sm/docs/*',
-            'junipernetworks/junos/.*',
-            'junipernetworks/junos/docs/*',
-            'junipernetworks/junos/tests/*',
-            'kubernetes/core/.*',
-            'kubernetes/core/docs/*',
-            'kubernetes/core/tests/*',
-            'lowlydba/sqlserver/.*',
-            'lowlydba/sqlserver/docs/*',
-            'lowlydba/sqlserver/tests/*',
-            'mellanox/onyx/.*',
-            'mellanox/onyx/tests/*',
-            'netapp/aws/.*',
-            'netapp/aws/tests/*',
-            'netapp/azure/.*',
-            'netapp/azure/tests/*',
-            'netapp/cloudmanager/.*',
-            'netapp/cloudmanager/tests/*',
-            'netapp/elementsw/.*',
-            'netapp/elementsw/tests/*',
-            'netapp/ontap/.*',
-            'netapp/ontap/tests/*',
-            'netapp/storagegrid/.*',
-            'netapp/storagegrid/tests/*',
-            'netapp/um_info/.*',
-            'netapp/um_info/tests/*',
-            'netapp_eseries/santricity/.*',
-            'netapp_eseries/santricity/tests/*',
-            'netbox/netbox/.*',
-            'netbox/netbox/docs/*',
-            'netbox/netbox/tests/*',
-            'ngine_io/cloudstack/.*',
-            'ngine_io/cloudstack/tests/*',
-            'ngine_io/exoscale/.*',
-            'ngine_io/exoscale/tests/*',
-            'ngine_io/vultr/.*',
-            'ngine_io/vultr/tests/*',
-            'openstack/cloud/docs/*',
-            'openvswitch/openvswitch/.*',
-            'openvswitch/openvswitch/docs/*',
-            'openvswitch/openvswitch/tests/*',
-            'ovirt/ovirt/.*',
-            'ovirt/ovirt/tests/*',
-            'purestorage/flasharray/.*',
-            'purestorage/flasharray/docs/*',
-            'purestorage/flasharray/tests/*',
-            'purestorage/flashblade/.*',
-            'purestorage/flashblade/tests/*',
-            'purestorage/fusion/.*',
-            'purestorage/fusion/tests/*',
-            'sensu/sensu_go/.*',
-            'sensu/sensu_go/docs/*',
-            'sensu/sensu_go/tests/*',
-            'splunk/es/.*',
-            'splunk/es/docs/*',
-            'splunk/es/tests/*',
-            't_systems_mms/icinga_director/.*',
-            't_systems_mms/icinga_director/docs/*',
-            'vmware/vmware_rest/.*',
-            'vmware/vmware_rest/docs/*',
-            'vmware/vmware_rest/tests/*',
-            'vultr/cloud/.*',
-            'vultr/cloud/tests/*',
-            'vyos/vyos/.*',
-            'vyos/vyos/docs/*',
-            'vyos/vyos/tests/*',
-            'wti/remote/docs/*',
+    packages=find_namespace_packages(
+        '.',
+        include=[
+            'ansible_collections*',
         ],
-    },
-    include_package_data=True,
+        exclude=[
+            'ansible_collections.*.*.tests*',
+            'ansible_collections.*.*.docs*',
+        ],
+    ),
+    package_dir={'': '.'},
+    package_data={'': ['*']},
     install_requires=[
         'ansible-core ~= 2.14.2',
     ],

[felixfontein]

@gotmax23 but does that include dotfiles in the root of a collection? And what happens to files or directories named docsfoo in the root of a collection? The former would be annoying, but ok, but the latter could be a problem.

[gotmax23]

but does that include dotfiles in the root of a collection?

No. * (from package_data={'': ['*']}) does not match dotfiles.

And what happens to files or directories named docsfoo in the root of a collection?

Files named docsfoo would be kept.

In any case, any changes we make here would need testing and should be delayed to Ansible 8 IMO.

[gotmax23]

I'd like to get this in for 8.0.0. Do you want to update your PR or should I submit a new one?

[felixfontein]

And what happens to files or directories named docsfoo in the root of a collection?

Files named docsfoo would be kept.

What about directories?

[felixfontein]

Just tested it: they are also kept. I modified the wrong file. They are actually not kept.

[felixfontein]

Also dotfiles in subdirectories are also not kept.

[felixfontein]

Changing

        exclude=[
            'ansible_collections.*.*.tests*',
            'ansible_collections.*.*.docs*',
        ],

to

        exclude=[
            'ansible_collections.*.*.tests.*',
            'ansible_collections.*.*.docs.*',
            'ansible_collections.*.*.tests',
            'ansible_collections.*.*.docs',
        ],

keeps subdirectories docsbar, but removes docs with all its subdirectories. With that, the only difference I can see are dotfiles in subdirectories.

[felixfontein]

    package_data={'': ['*', '*/*/*/**/.*']},

ensures that dotfiles in collection roots are gone, but dotfiles in subdirectories are kept.

[felixfontein]

I created a new PR: #502

Since most changes to #434 are not needed I don't think it's worth updating that one.