search

ansible-collections / community.general

Ansible Community General Collection
https://galaxy.ansible.com/ui/repo/published/community/general/
GNU General Public License v3.0
823 stars 1.52k forks source link

xml Module Failure with lxml.etree #7171

Open shepster opened 1 year ago

shepster commented 1 year ago

Summary

Running the Ansible community.general.xml module on a playbook that worked in the past now throws an error.

Issue Type

Bug Report

Component Name

xml

Ansible Version

$ ansible --version
ansible [core 2.15.3]
  config file = None
  configured module search path = ['/Users/shepherd/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/8.3.0/libexec/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/shepherd/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.4 (main, Jul 25 2023, 17:08:27) [Clang 14.0.0 (clang-1400.0.29.202)] (/usr/local/Cellar/ansible/8.3.0/libexec/bin/python)
  jinja version = 3.1.2
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general
# /usr/local/Cellar/ansible/8.3.0/libexec/lib/python3.11/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 7.3.0

Configuration

$ ansible-config dump --only-changed
CONFIG_FILE() = None

The remote version of the lxml library is:

[root@128.34.171.254 ~]# pip list | grep lxml
lxml                             3.2.1
[root@128.34.171.254 ~]#

OS / Environment

local system: macOS Monterey 12.6.8 target system: Amazon Linux 2

Steps to Reproduce

---

- name: Configure QA Services / Jira Nodes
  hosts: jira_qa
  remote_user: ec2-user
  become: true
  become_user: root
  become_method: sudo
  gather_facts: false

  vars:
#     ansible_python_interpreter: /usr/bin/python
#     ansible_python_interpreter: /usr/bin/python3
#     ansible_python_interpreter: /usr/bin/python3.8

  tasks:
    - name: "Add FIPSMode attribute in server.xml"
      community.general.xml:
        path: "/opt/atlassian/jira-software/current/conf/server.xml"
        xpath: /Server/Listener[@className='org.apache.catalina.core.AprLifecycleListener']
        attribute: FIPSMode
        value: off

...

Expected Results

I expected the Ansible code to run and not error out. I have run this code before and it has worked. I don't know what changed in my environment. I think it was a macOS Monterey upgrade 12.6.7 -> 12.6.8.

My example is using Python 2.7 on the remote system. I have also tried it with # ansible_python_interpreter: /usr/bin/python3 and ansible_python_interpreter: /usr/bin/python3.8 with similar results.

Actual Results

shepherd@duffman dc-deployments-automation % ansible-playbook -v -i inv/hosts xmlTest.yml
Using /Users/shepherd/prog/dc-deployments-automation/ansible.cfg as config file

PLAY [Configure QA Services / Jira Nodes] **************************************************************************

TASK [Add FIPSMode attribute in server.xml] ************************************************************************
Tuesday 29 August 2023  10:15:11 -0700 (0:00:00.079)       0:00:00.079 ********
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: Argument must be bytes or unicode, got 'bool'
fatal: [128.34.171.254]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 128.34.171.254 closed.
", "module_stdout": "Traceback (most recent call last):
  File \"/home/ec2-user/.ansible/tmp/ansible-tmp-1693329313.416832-52117-179003560861876/AnsiballZ_xml.py\", line 107, in <module>
    _ansiballz_main()
  File \"/home/ec2-user/.ansible/tmp/ansible-tmp-1693329313.416832-52117-179003560861876/AnsiballZ_xml.py\", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File \"/home/ec2-user/.ansible/tmp/ansible-tmp-1693329313.416832-52117-179003560861876/AnsiballZ_xml.py\", line 48, in invoke_module
    run_name='__main__', alter_sys=True)
  File \"/usr/lib64/python2.7/runpy.py\", line 188, in run_module
    fname, loader, pkg_name)
  File \"/usr/lib64/python2.7/runpy.py\", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code
    exec code in run_globals
  File \"/tmp/ansible_community.general.xml_payload_5xUqf9/ansible_community.general.xml_payload.zip/ansible_collections/community/general/plugins/modules/xml.py\", line 996, in <module>
  File \"/tmp/ansible_community.general.xml_payload_5xUqf9/ansible_community.general.xml_payload.zip/ansible_collections/community/general/plugins/modules/xml.py\", line 982, in main
  File \"/tmp/ansible_community.general.xml_payload_5xUqf9/ansible_community.general.xml_payload.zip/ansible_collections/community/general/plugins/modules/xml.py\", line 707, in set_target
  File \"/tmp/ansible_community.general.xml_payload_5xUqf9/ansible_community.general.xml_payload.zip/ansible_collections/community/general/plugins/modules/xml.py\", line 701, in set_target_inner
  File \"src/lxml/lxml.etree.pyx\", line 713, in lxml.etree._Element.set
  File \"src/lxml/apihelpers.pxi\", line 520, in lxml.etree._setAttributeValue
  File \"src/lxml/apihelpers.pxi\", line 1333, in lxml.etree._utf8
TypeError: Argument must be bytes or unicode, got 'bool'
", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

PLAY RECAP *********************************************************************************************************
128.34.171.254             : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Tuesday 29 August 2023  10:15:17 -0700 (0:00:05.734)       0:00:05.814 ********
===============================================================================
Add FIPSMode attribute in server.xml ------------------------------------------------------------------------ 5.73s
shepherd@duffman dc-deployments-automation %

Code of Conduct

ansibullbot commented 1 year ago

Files identified in the description:

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot commented 1 year ago

cc @cmprescott @dagwieers @sm4rk0 @tbielawa click here for bot help

felixfontein commented 1 year ago

You are providing a boolean (off) as the value, which the module doesn't like. Better provide the value as a string: value: "off".

(It shoudn't crash though, but result in a better error message.)

shepster commented 1 year ago

This bug is something that is preventing me from running my scripts. It is something that was not an issue a couple of weeks ago. I'm not sure what I did that updated my system, though I suspect it was an upgrade from macOS Monterey 12.6.7 -> 12.6.8. If I wanted to regress my system, should I try going back to an older version of community.general or should I regress Ansible?

felixfontein commented 1 year ago

The only code change to this module since November 2022 (then it was moved to another location, and GitHub's history is unable to show what happened before without lots of manual work) was 6ec049734e4b54edcf8dffbad64bf777fc9701e6, which is very unlikely to be related to this. So either you did upgrade from a rather old version of community.general or Ansible to a newer one, or this is not related to a change in Ansible or community.general. This could also have been caused by a newer version of lxml (on the remote system!), for example; there has been a new lxml release on July 5th, 2023.

Unfortunately the project homepage claims the latest version is 2.9.2 (released in December 2022), and only has its changelog. The 2.9.3 changelog in the Git repo (https://github.com/lxml/lxml/blob/master/CHANGES.txt#L51-L73) doesn't contain something obvious that could have caused this.

Hmm, I decided to look a bit at https://github.com/lxml/lxml/blob/master/src/lxml/apihelpers.pxi - your error message says that the error was raised in line 1333. The last time that error was raised in line 1333 of that file was https://github.com/lxml/lxml/blob/f97698736299d44769d23fe6e7b4456e82d1c132/src/lxml/apihelpers.pxi, which was there until July 2013 (https://github.com/lxml/lxml/commit/6d62f10a3bbe9d74fb5ae9b693bfd5ac2a9aa0e7). So you seem to have a very ancient version of lxml installed - which is most likely the reason why this doesn't work.

shepster commented 1 year ago

I appreciate your investigation into this matter further. As I understand it, lxml is a library being used on the remote machine and as I noted in the configuration above, it is 3.2.1 (due to python-lxml.x86_64 3.2.1-4.amzn2.0.4 @amzn2-core). I've done a pip install lxml before, but it turns out a pip install --upgrade lxml is needed to advance the version to 4.9.3. Sadly, with this upgraded version of lxml I am getting a similar message:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: Argument must be bytes or unicode, got 'bool'
fatal: [bossjiraqa5]: FAILED! => {"changed": false, "module_stderr": "Shared connection to bossjiraqa5 closed.
", "module_stdout": "Traceback (most recent call last):
  File \"/home/ec2-user/.ansible/tmp/ansible-tmp-1693623901.7250628-99322-227836736248396/AnsiballZ_xml.py\", line 107, in <module>
    _ansiballz_main()
  File \"/home/ec2-user/.ansible/tmp/ansible-tmp-1693623901.7250628-99322-227836736248396/AnsiballZ_xml.py\", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File \"/home/ec2-user/.ansible/tmp/ansible-tmp-1693623901.7250628-99322-227836736248396/AnsiballZ_xml.py\", line 48, in invoke_module
    run_name='__main__', alter_sys=True)
  File \"/usr/lib64/python2.7/runpy.py\", line 188, in run_module
    fname, loader, pkg_name)
  File \"/usr/lib64/python2.7/runpy.py\", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code
    exec code in run_globals
  File \"/tmp/ansible_community.general.xml_payload_Qwx3Hi/ansible_community.general.xml_payload.zip/ansible_collections/community/general/plugins/modules/xml.py\", line 996, in <module>
  File \"/tmp/ansible_community.general.xml_payload_Qwx3Hi/ansible_community.general.xml_payload.zip/ansible_collections/community/general/plugins/modules/xml.py\", line 982, in main
  File \"/tmp/ansible_community.general.xml_payload_Qwx3Hi/ansible_community.general.xml_payload.zip/ansible_collections/community/general/plugins/modules/xml.py\", line 707, in set_target
  File \"/tmp/ansible_community.general.xml_payload_Qwx3Hi/ansible_community.general.xml_payload.zip/ansible_collections/community/general/plugins/modules/xml.py\", line 701, in set_target_inner
  File \"src/lxml/etree.pyx\", line 834, in lxml.etree._Element.set
  File \"src/lxml/apihelpers.pxi\", line 594, in lxml.etree._setAttributeValue
  File \"src/lxml/apihelpers.pxi\", line 1539, in lxml.etree._utf8
TypeError: Argument must be bytes or unicode, got 'bool'
", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
felixfontein commented 1 year ago

My guess is that before, you were using an even older version of lxml, that did that conversion automatically, and that they changed that behavior a long time ago (10 years or more) and it stayed the same since then.

The module documentation also states that you have to provide a string or none, but doesn't say that a boolean is valid input. I guess it should check for that, but right now it does not. So you have to make sure you are passing a string and not a boolean.