search

ansible-collections / community.crypto

The community.crypto collection for Ansible.
https://galaxy.ansible.com/ui/repo/published/community/crypto/
Other
98 stars 89 forks source link

[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at 0x7f5705344c10>): 'utf-8' codec can't encode character '\udc80' in position 1731: surrogates not allowed #654

Open cidrbl0ck opened 1 year ago

cidrbl0ck commented 1 year ago

Greetings: So to start off I do know there are pre-existing Issues on this exact thing. To be perfeclty honest I was not sure the best course of action.. to re-open one of those to to create a new issue so apologies if I chose poorly.

I'm fighting the same issue and while I've found multiple issues on this problem I do not understand what we should be doing Ansible Core issue 80258 gives the impression that the community.crypto modules would eventually return utf-8 encoded data.

However Crypto Issue 591 is entirely confusing and I'm left feeling that this problem will never be resolved and the module is just broke.

I hope someone corrects me here because it's using this module or stringing ugly shell commands for openssl s_client together....

A very small sub-issue to this that I don't know if related to the UTF-8 issue or not would be the type_debug. The docs page for this module states that note_after is a string. So getting AnsibleUnsafeText is confusing as well.

ISSUE TYPE
COMPONENT NAME

community.crypto.get_certificate

ANSIBLE VERSION
ansible [core 2.14.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/svc-ansiblemgmt/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/svc-ansiblemgmt/.local/lib/python3.9/site-packages/ansible
  ansible collection location = /home/svc-ansiblemgmt/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.5 (default, Nov 23 2021, 15:27:38) [GCC 9.3.0] (/usr/bin/python3.9)
  jinja version = 3.1.2
  libyaml = False
COLLECTION VERSION
# /home/svc-ansiblemgmt/.local/lib/python3.9/site-packages/ansible_collections
Collection       Version
---------------- -------
community.crypto 2.15.1

# /home/svc-ansiblemgmt/.ansible/collections/ansible_collections
Collection       Version
---------------- -------
community.crypto 2.15.1
CONFIGURATION
ANSIBLE_COW_SELECTION(/etc/ansible/ansible.cfg) = default
ANSIBLE_NOCOWS(/etc/ansible/ansible.cfg) = True
CONFIG_FILE() = /etc/ansible/ansible.cfg
DEFAULT_CALLBACK_PLUGIN_PATH(/etc/ansible/ansible.cfg) = ['/usr/share/ansible/plugins/callback']
DEFAULT_INVENTORY_PLUGIN_PATH(/etc/ansible/ansible.cfg) = ['/usr/share/ansible/plugins/inventory']
DEFAULT_MANAGED_STR(/etc/ansible/ansible.cfg) = Ansible managed by {host}
GALAXY_SERVER_LIST(/etc/ansible/ansible.cfg) = ['aaphub', 'galaxy']
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
INVENTORY_IGNORE_EXTS(/etc/ansible/ansible.cfg) = ['~', '.orig', '.bak', '.cfg', '.retry', '.pyc', '.pyo']
PARAMIKO_HOST_KEY_AUTO_ADD(/etc/ansible/ansible.cfg) = True
PARAMIKO_LOOK_FOR_KEYS(/etc/ansible/ansible.cfg) = False
OS / ENVIRONMENT

Ubuntu 20.04 LTS

STEPS TO REPRODUCE
    - name: Poll target SSL cert for expiration date
      community.crypto.get_certificate:
        host: "{{ site_url }}"
        port: "{{ host_port | default('443') }}"
      run_once: true
      register: expire_check
      delegate_to: localhost
      tags: check

    - name: debug
      debug:
        msg: "{{ expire_check }}"
      tags: check

    - name: Set not_after fact
      ansible.builtin.set_fact:
        exp_zulu: '{{ expire_check.not_after }}'
      tags: check

    - name: Return expiration data
      ansible.builtin.debug:
        var: exp_zulu | type_debug
      tags: check

    - name: Set Host fact - Cert expiration
      ansible.builtin.set_fact:
        ssl_expiration: "{{ ( exp_zulu | to_datetime).iso8601() }}"
      tags: check
EXPECTED RESULTS
ACTUAL RESULTS
TASK [Poll target SSL cert for expiration date] *************************************************************************************************************
task path: /etc/ansible/new_devel/sysops-aap/aap-get-webserver.yml:68
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: svc-ansible
<localhost> EXEC /bin/sh -c 'echo ~svc-ansiblemgmt && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/svc-ansible/.ansible/tmp `"&& mkdir "` echo /home/svc-ansible/.ansible/tmp/ansible-tmp-1694190402.4776316-1205021-96630419123871 `" && echo ansible-tmp-1694190402.4776316-1205021-96630419123871="` echo /home/svc-ansible/.ansible/tmp/ansible-tmp-1694190402.4776316-1205021-96630419123871 `" ) && sleep 0'
Using module file /home/svc-ansible/.ansible/collections/ansible_collections/community/crypto/plugins/modules/get_certificate.py
<localhost> PUT /home/svc-ansible/.ansible/tmp/ansible-local-1204968ytru3v9a/tmp6ga3i3r2 TO /home/svc-ansible/.ansible/tmp/ansible-tmp-1694190402.4776316-1205021-96630419123871/AnsiballZ_get_certificate.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/svc-ansible/.ansible/tmp/ansible-tmp-1694190402.4776316-1205021-96630419123871/ /home/svc-ansible/.ansible/tmp/ansible-tmp-1694190402.4776316-1205021-96630419123871/AnsiballZ_get_certificate.py && sleep 0'
<localhost> EXEC /bin/sh -c 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-gptvnubspqzzabaukgftlgdismhluqet ; /usr/bin/python3.9 /home/svc-ansible/.ansible/tmp/ansible-tmp-1694190402.4776316-1205021-96630419123871/AnsiballZ_get_certificate.py'"'"' && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/svc-ansible/.ansible/tmp/ansible-tmp-1694190402.4776316-1205021-96630419123871/ > /dev/null 2>&1 && sleep 0'
[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at 0x7f853a23ef10>): 'utf-8'
codec can't encode character '\udc80' in position 1731: surrogates not allowed
Callback Exception:
  File "/home/svc-ansible/.local/lib/python3.9/site-packages/ansible/executor/task_queue_manager.py", line 450, in send_callback
    method(*new_args, **kwargs)
   File "/home/svc-ansible/.local/lib/python3.9/site-packages/ansible/plugins/callback/default.py", line 106, in v2_runner_on_ok
    self._display.display(msg, color=color)
   File "/home/svc-ansible/.local/lib/python3.9/site-packages/ansible/utils/display.py", line 289, in display
    fileobj.write(msg2)

TASK [debug] ************************************************************************************************************************************************
task path: /etc/ansible/new_devel/sysops-aap/aap-get-webserver.yml:77
[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at 0x7f853a23ef10>): 'utf-8'
codec can't encode character '\udc80' in position 1766: surrogates not allowed
Callback Exception:
  File "/home/svc-ansible/.local/lib/python3.9/site-packages/ansible/executor/task_queue_manager.py", line 450, in send_callback
    method(*new_args, **kwargs)
   File "/home/svc-ansible/.local/lib/python3.9/site-packages/ansible/plugins/callback/default.py", line 106, in v2_runner_on_ok
    self._display.display(msg, color=color)
   File "/home/svc-ansible/.local/lib/python3.9/site-packages/ansible/utils/display.py", line 289, in display
    fileobj.write(msg2)

TASK [Set not_after fact] ***********************************************************************************************************************************
task path: /etc/ansible/new_devel/sysops-aap/aap-get-webserver.yml:82
ok: [quickdnsd01] => {
    "ansible_facts": {
        "exp_zulu": "20231205235959Z"
    },
    "changed": false
}

TASK [Return expiration data] *******************************************************************************************************************************
task path: /etc/ansible/new_devel/sysops-aap/aap-get-webserver.yml:87
ok: [quickdnsd01] => {
    "exp_zulu | type_debug": "AnsibleUnsafeText"
}

TASK [Set Host fact - Cert expiration] **********************************************************************************************************************
task path: /etc/ansible/new_devel/sysops-aap/aap-get-webserver.yml:94
fatal: [quickdnsd01]: FAILED! => {
    "msg": "the field 'args' has an invalid value ({'ssl_expiration': '{{ ( exp_zulu | to_datetime).iso8601() }}'}), and could not be converted to an dict.The error was: time data '20231205235959Z' does not match format '%Y-%m-%d %H:%M:%S'\n\nThe error appears to be in '/etc/ansible/new_devel/sysops-aap/aap-get-webserver.yml': line 94, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: Set Host fact - Cert expiration\n      ^ here\n"
}
felixfontein commented 1 year ago

You can simply add asn1_base64: true to community.crypto.get_certificate's options.

That will be the new default of asn1_base64 in community.crypto 3.0.0 anyway, but the 3.0.0 release won't happen so soon, until then you can manually set it to true.

A very small sub-issue to this that I don't know if related to the UTF-8 issue or not would be the type_debug. The docs page for this module states that not_after is a string. So getting AnsibleUnsafeText is confusing as well.

AnsibleUnsafeText is an ansible-core internal of a string returned by a module (it's marked 'unsafe' because of that, so it isn't automatically templated - that would allow modules executed on remote targets to execute code on the controller, which would be very bad from a security point of view).

The problem you have in the last set_fact is that to_datetime expects a very specific date/time format, which is very different from the date/time format returned by the module. 20231205235959Z is in ASN.1 UTCTime format (https://obj-sys.com/asn1tutorial/node15.html), while to_datetime apparently expects %Y-%m-%d %H:%M:%S (with dashes, space, and colons, and no Z).

cidrbl0ck commented 1 year ago

Thanks Felix, I did see asn1_base64: true mentioned but the doc page wasn't clear on how to make use of that. I did attempt to add it into the task.....

And well wouldn't you know it, it worked this time. SMH.

No so before I posted the Issue I did attempt the task as:

    - name: Poll target SSL cert for expiration date
      #ansible.builtin.shell:
        #cmd: openssl s_client -connect {{ site_url }}:443 | openssl x509 -noout -dates
      community.crypto.get_certificate:
        host: "{{ site_url }}"
        port: "{{ host_port | default('443') }}"
        asn1_base64: true
      #run_once: true
      register: expire_check
      delegate_to: localhost
      tags: check

Yet it errored out, but in true self embarrassing fashion it worked this time. Well I guess disregard this :)

I won't lie, the UnsafeText bit is still super murky but it doesn't matter now because I'm pulling Zulu time correctly. Thanks for the help!

felixfontein commented 1 year ago

Glad to hear that it now works :)