Current version pushed to Ansible Galaxy has typo in `/netapp/ontap/plugins/modules/na_ontap_s3_users.py`

abegosum commented 7 months ago

Summary

When installing either through the Ansible ppa package or the ansible-galaxy command, the na_ontap_s3_users.py file has the word git before the shebang.

git #!/usr/bin/python

# (c) 2022, NetApp, Inc
# GNU General Public License v3.0+
# (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

from __future__ import absolute_import, division, print_function

__metaclass__ = type

DOCUMENTATION = '''
module: na_ontap_s3_users

Removing this is necessary to make an install from a package complete or to avoid errors when using this module. On my system, this file is installed, by default, to /usr/lib/python3/dist-packages/ansible_collections/netapp/ontap/plugins/modules/na_ontap_s3_users.py.

Component Name

netapp.ontap.na_ontap_s3_users

Ansible Version

$ ansible --version
ansible [core 2.15.6]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True

ONTAP Collection Version

$ ansible-galaxy collection list

# /usr/lib/python3/dist-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    6.5.0  
ansible.netcommon             5.3.0  
ansible.posix                 1.5.4  
ansible.utils                 2.11.0 
ansible.windows               1.14.0 
arista.eos                    6.2.1  
awx.awx                       22.7.0 
azure.azcollection            1.19.0 
check_point.mgmt              5.1.1  
chocolatey.chocolatey         1.5.1  
cisco.aci                     2.8.0  
cisco.asa                     4.0.3  
cisco.dnac                    6.7.6  
cisco.intersight              1.0.27 
cisco.ios                     4.6.1  
cisco.iosxr                   5.0.3  
cisco.ise                     2.5.16 
cisco.meraki                  2.16.13
cisco.mso                     2.5.0  
cisco.nso                     1.0.3  
cisco.nxos                    4.4.0  
cisco.ucs                     1.10.0 
cloud.common                  2.1.4  
cloudscale_ch.cloud           2.3.1  
community.aws                 6.4.0  
community.azure               2.0.0  
community.ciscosmb            1.0.7  
community.crypto              2.16.0 
community.digitalocean        1.24.0 
community.dns                 2.6.3  
community.docker              3.4.10 
community.fortios             1.0.0  
community.general             7.5.1  
community.google              1.0.0  
community.grafana             1.6.1  
community.hashi_vault         5.0.1  
community.hrobot              1.8.1  
community.libvirt             1.3.0  
community.mongodb             1.6.3  
community.mysql               3.8.0  
community.network             5.0.2  
community.okd                 2.3.0  
community.postgresql          2.4.3  
community.proxysql            1.5.1  
community.rabbitmq            1.2.3  
community.routeros            2.10.0 
community.sap                 1.0.0  
community.sap_libs            1.4.1  
community.skydive             1.0.0  
community.sops                1.6.7  
community.vmware              3.11.1 
community.windows             1.13.0 
community.zabbix              2.1.0  
containers.podman             1.11.0 
cyberark.conjur               1.2.2  
cyberark.pas                  1.0.23 
dellemc.enterprise_sonic      2.2.0  
dellemc.openmanage            7.6.1  
dellemc.powerflex             1.9.0  
dellemc.unity                 1.7.1  
f5networks.f5_modules         1.27.0 
fortinet.fortimanager         2.3.0  
fortinet.fortios              2.3.4  
frr.frr                       2.0.2  
gluster.gluster               1.0.2  
google.cloud                  1.2.0  
grafana.grafana               2.2.3  
hetzner.hcloud                1.16.0 
hpe.nimble                    1.1.4  
ibm.qradar                    2.1.0  
ibm.spectrum_virtualize       1.12.0 
ibm.storage_virtualize        2.1.0  
infinidat.infinibox           1.3.12 
infoblox.nios_modules         1.5.0  
inspur.ispim                  1.3.0  
inspur.sm                     2.3.0  
junipernetworks.junos         5.3.0  
kubernetes.core               2.4.0  
lowlydba.sqlserver            2.2.2  
microsoft.ad                  1.3.0  
netapp.aws                    21.7.1 
netapp.azure                  21.10.1
netapp.cloudmanager           21.22.1
netapp.elementsw              21.7.0 
netapp.ontap                  22.8.1 
netapp.storagegrid            21.11.1
netapp.um_info                21.8.1 
netapp_eseries.santricity     1.4.0  
netbox.netbox                 3.15.0 
ngine_io.cloudstack           2.3.0  
ngine_io.exoscale             1.1.0  
ngine_io.vultr                1.1.3  
openstack.cloud               2.1.0  
openvswitch.openvswitch       2.1.1  
ovirt.ovirt                   3.2.0  
purestorage.flasharray        1.21.0 
purestorage.flashblade        1.14.0 
purestorage.fusion            1.6.0  
sensu.sensu_go                1.14.0 
servicenow.servicenow         1.0.6  
splunk.es                     2.1.0  
t_systems_mms.icinga_director 1.33.1 
telekom_mms.icinga_director   1.34.1 
theforeman.foreman            3.14.0 
vmware.vmware_rest            2.3.1  
vultr.cloud                   1.10.0 
vyos.vyos                     4.1.0  
wti.remote                    1.0.5

ONTAP Version

N/A - error happens on install of Ansible

Playbook

N/A - error happens on install of Ansible

Steps to Reproduce

On Ubuntu Jammy (22.04.3 in my case) with the Ansible official PPA enabled, attempt to install or upgrade the ansible package.

Expected Results

I expected the default collections to install alongside ansible and configure appropriately, including the netapp.ontap collection.

Actual Results

Upon install or upgrade, a syntax error in the netapp.ontap module causes the install to break.

$ apt upgrade -y
--- output cut ---
Setting up ansible (8.6.0-1ppa~jammy) ...
  File "/usr/lib/python3/dist-packages/ansible_collections/netapp/ontap/plugins/modules/na_ontap_s3_users.py", line 7
    from __future__ import absolute_import, division, print_function
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: from __future__ imports must occur at the beginning of the file
dpkg: error processing package ansible (--configure):
 installed ansible package post-installation script subprocess returned error exit status 1

abegosum commented 7 months ago

Another note on recreation of the issue- you can install the collection directly from ansible-galaxy into a test directory and the same typo will appear.

# ansible-galaxy collection install --force -p ./anstest netapp.ontap
Starting galaxy collection install process
[WARNING]: The specified collections path '/root/anstest' is not part of the configured Ansible collections paths
'/root/.ansible/collections:/usr/share/ansible/collections'. The installed collection will not be picked up in an Ansible run, unless within a playbook-
adjacent collections directory.
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/netapp-ontap-22.8.1.tar.gz to /root/.ansible/tmp/ansible-local-93123mqb2qvxe/tmph7np_ld7/netapp-ontap-22.8.1-ju5on8hg
Installing 'netapp.ontap:22.8.1' to '/root/anstest/ansible_collections/netapp/ontap'
netapp.ontap:22.8.1 was installed successfully

# head anstest/ansible_collections/netapp/ontap/plugins/modules/na_ontap_s3_users.py 
git #!/usr/bin/python

# (c) 2022, NetApp, Inc
# GNU General Public License v3.0+
# (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

from __future__ import absolute_import, division, print_function

__metaclass__ = type

adosztal commented 7 months ago

I just ran into this too when upgrading Ansible to 8.6.0 on Ubuntu 22.04, got the same result. Manually removing git before the shebang of /usr/lib/python3/dist-packages/ansible_collections/netapp/ontap/plugins/modules/na_ontap_s3_users.py has solved the issue.

maxkratz commented 7 months ago

I can confirm this on multiple systems. Removing the String git in the first line resolves the issue (at least for me).

rit001 commented 7 months ago

My Canary deployment system is showing this as a 'show-stopper' for our environment if I were to deploy any real systems.

Typos into the wrong window are a common issue, so this is a minor error, that is showing a limitation of whatever CI process is currently in place for the repo.

Just to be clear this seems to be an error that is part of the Ansible 8.6.0 release as I can not find the error within this repo or recent past releases.

egut commented 7 months ago

The main branch in this repo don't seams to not have the issues https://github.com/ansible-collections/netapp.ontap/blob/main/plugins/modules/na_ontap_s3_users.py

This leads my to assume one (or both) of the following:

The person that did the push typed "git" in the local files from his/hers computer.
The issue is in bitbucket where this code is copied from (I assume so by the latest git comments: "Sync bitbucket and GitHub" by @carchi8py )

Ether way a new release is appreciated :)

rit001 commented 7 months ago

Opened as an issue over at https://github.com/ansible-community/ppa/issues/77

sivel commented 7 months ago

Thank you for your interest in Ansible. Unfortunately the maintainers of this repo do not control either the netapp.ontapp collection, nor the ansible package. This repository is specifically for the ansible-core package.

As I see above, there is already an issue open for the netapp.ontapp repository, but it looks like there are some requests to have a new ansible package released. You will need to consult the maintainers of the ansible package at https://github.com/ansible-community/ansible-build-data or through the community communication mechanisms listed at https://docs.ansible.com/ansible/latest/community/communication.html

adosztal commented 7 months ago

Ffs, this repo is called "netapp.ontap"! What are you doing here then? And why the hell does the "Issue tracker" on the collection's page point here?

The problem wasn't in the Ansible package, we just can't install it because of the issue of the ontap collection!

sivel commented 7 months ago

I apologize, that's my mistake, I got the tabs mixed up.

rit001 commented 7 months ago

OK, while people are getting their wire's crossed there is a larger issue - where on Earth has the building of the Ansible distribution pulled in this typo?

This repo does not seem to record the change to the file that is now causing the issue with the final 8.6.0 Ansible distribution. So is anyone understands the workflow that connects these 2 independent projects could they take a look around?

lattwood commented 7 months ago

https://galaxy.ansible.com/ui/repo/published/netapp/ontap/docs/ links to here for the Issue Tracker.

I question why the netapp collection is included by default.

So uh, does this mean this is built and pushed from a local developer machine?

sivel commented 7 months ago

So uh, does this mean this is built and pushed from a local developer machine?

I'm guessing so. The repo doesn't contain that, but it's in the artifact pushed to galaxy. So it would seem the local repo is "dirty" and was used in that state to build the artifact from.

lattwood commented 7 months ago

Heh, since I have your attention @sivel, are there any kind of build/release standards/requirements for inclusion in default collections? If not, should there be, in light of a pushed artifact that was apparently built from a dirty tree?

It does make me wonder what other things are dirty in that collection.

rit001 commented 7 months ago

When following the build process for ppa distributions I ended up at the following

https://github.com/ansible-community/ppa/issues/77

So people are asking the same questions, in the right place :)

The build process seems to be github driven, so something went badly wrong to end up with 'git ' being added to a random file.

lattwood commented 7 months ago

@rit001 nope, the artifact on Ansible Galaxy has the git addition as well.

lattwood commented 7 months ago

And the tests for this collection have been failing for quite some time.

https://github.com/ansible-collections/netapp.ontap/actions/workflows/main.yml

If you drill down into individual test runs, it shows they've been failing on 2.15 which is used in ansible 8.6- https://github.com/ansible-community/ansible-build-data/blob/f4f81a908d6bc4f92212d62282ad49759a157f93/8/ansible-8.6.0.deps#L2

sivel commented 7 months ago

@lattwood unfortunately I cannot really help there. While I'm a dev for ansible-core, I have no real interaction with the packaging of the ansible package. There were sanity tests that ran during the process of creating the ansible package, but I have no current knowledge of whether that is still done. It would have been caught by ansible-test, at minimum by the validate-modules or the import tests:

ERROR: plugins/modules/na_ontap_s3_users.py:0:0: import-error: Exception attempting to import module for argument_spec introspection, 'from __future__ imports must occur at the beginning of the file (na_ontap_s3_users.py, line 7)'
ERROR: plugins/modules/na_ontap_s3_users.py:7:0: traceback: SyntaxError: from __future__ imports must occur at the beginning of the file

lattwood commented 7 months ago

Well, here's one thing to be relieved over. This is the only file that was impacted.

carchi8py commented 7 months ago

I'll get a fix out for this today.

carchi8py commented 7 months ago

I'm going to fix this directly in Github. As i don't see this in our internal version.

Should have the fix out on galaxy in a few minutes Screenshot 2023-11-09 at 9 46 09 AM

carchi8py commented 7 months ago

I pushed the fix out and it on Galaxy.

https://galaxy.ansible.com/ui/repo/staging/netapp/ontap/?version=22.8.2

It looks like a file was touched in pipeline release git repo. The change was never checked in to either our bitbucket or github repo.

I"ll modify our internal pipeline to do a fresh git clone from now when building new releases.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   ansible_collections/netapp/ontap/plugins/modules/na_ontap_s3_users.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    ansible_collections/netapp/ontap/netapp-ontap-21.10.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.11.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.12.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.13.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.13.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.14.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.14.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.15.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.15.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.16.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.17.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.17.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.17.2.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.17.3.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.18.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.18.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.19.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.19.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.20.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.22.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.23.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.24.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.24.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.6.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.6.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.7.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.8.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.8.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-21.9.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.0.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.0.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.1.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.2.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.3.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.4.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.4.1.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.5.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.6.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.7.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.8.0.tar.gz
    ansible_collections/netapp/ontap/netapp-ontap-22.8.1.tar.gz

no changes added to commit (use "git add" and/or "git commit -a")

rit001 commented 7 months ago

Are you able to cause the ppa process to rebuild, or are you able to contact the people who can?

carchi8py commented 7 months ago

@rit001 i'm not able to but i posted the same information in https://github.com/ansible-community/ppa/issues/77 so I'm hoping someone on their side can update it to the latest which should fix the issue.

oraNod commented 7 months ago

I pushed the fix out and it on Galaxy.

https://galaxy.ansible.com/ui/repo/staging/netapp/ontap/?version=22.8.2

@carchi8py Hi, thank you for this. There is talk of cutting an Ansible 8.6.1 release to pick up this fix but 22.8.2 isn't tagged in the repository. Would you be able to create that?

carchi8py commented 7 months ago

@oraNod I'll have it tags in github once the change makes it thought our bitbucket pipeline (last steps is to push to githuib). Should be here with in the hour

carchi8py commented 7 months ago

Since it a super minor change i'm going to manually do it here and get the tag up (give me a few minutes)

carchi8py commented 7 months ago

Ok 22.8.2 release is on github

oraNod commented 7 months ago

Thanks @carchi8py

gotmax23 commented 7 months ago

Ansible 8.6.1 has been released. It contains the netapp.ontap 2.8.2 update that fixes this issue.

dray92 commented 7 months ago

@carchi8py, firstly thanks for your help in getting this resolved.

Couple of questions since https://github.com/ansible-community/ppa/issues/77 is now closed:

when will the apt repos get populated w/ 8.6.1? to get unblocked immediately, this is what we're still using (for apt), which works either when running on host, VM or containers:

apt-get update && apt-get install -y ansible \
|| ( \
# If the installation fails, apply the workaround:
# Comment out the first line of the na_ontap_s3_users.py script
# Failed to install Ansible, try workaround based on
# https://github.com/ansible-community/ppa/issues/77#issuecomment-1802847056
sed -i -e '1s/^[^#]*//' /usr/lib/python3/dist-packages/ansible_collections/netapp/ontap/plugins/modules/na_ontap_s3_users.py && \
# Then try to fix any broken dependencies
apt-get install -y --fix-broken \
)

This software runs in various companies/enterprises. As such, it doesn't inspire confidence when a release contains an error which isn't even in source tree (indicates lack of centralized/controlled build system). Is there a github issue (or issue tracker) that we can follow to make sure that a centralized system for deploying updates is enforced (and that developers can't push out updates from arbitrary environments)?

maxkratz commented 7 months ago

when will the apt repos get populated w/ 8.6.1?

On my systems (Debian-based, using the PPA), the error was gone as of today morning.

rit001 commented 7 months ago

" * This software runs in various companies/enterprises. As such, it doesn't inspire confidence when a release contains an error which isn't even in source tree (indicates lack of centralized/controlled build system). Is there a github issue (or issue tracker) that we can follow to make sure that a centralized system for deploying updates is enforced (and that developers can't push out updates from arbitrary environments)?"

This was answered in

  https://github.com/ansible-collections/netapp.ontap/issues/190

with the following statement from carchi8py

"I'll modify our internal pipeline to do a fresh git clone from now when building new releases."

carchi8py commented 7 months ago

This was fixed in 22.8.2

ansible-collections / netapp.ontap