[Bug] [ansible] SyntaxError: Non-UTF-8 code starting with '\\x88' in file <stdin> on line 1, but no encoding declared

gngrossi commented 2 months ago

Is there an existing issue for this?

[X] There are no existing issues.

Bug description

Issue running an ansible command for the ping module. Able to run a playbook for it.

IBM z/OS Ansible core Version

v1.9.0

IBM Z Open Automation Utilities

v1.2.5 (default)

IBM Enterprise Python

v3.12.x

ansible-version

v2.16.x (default)

z/OS version

v2.5 (default)

Ansible module

No response

Playbook verbosity output.

bash-4.4$ ansible -m ping all mvs-sysd | FAILED! => { "changed": false, "module_stderr": "SyntaxError: Non-UTF-8 code starting with '\x88' in file on line 1, but no encoding declared; see https://peps.python.org/pep-0263/ for details\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 }

playbook runs successfully.

hosts: MA1 collections:
- ibm.ibm_zos_core gather_facts: false environment: "{{environment_vars}}" tasks:
- name: Pinging host - {{inventory_hostname}} ping:
  zos_ping:
  
  register: result
- name: Response debug: msg: "{{result.ping}}" #

Ansible configuration.

defaults]
forks = 25
inventory = ~/playbooks/inventory.yml
host_key_checking = false
remote_port = 2222
timeout = 15
#deprecation_warnings=false

[ssh_connection]
pipelining = true

Contents of the inventory

MA1:
  hosts:
    mvs-sysd:
  vars:
    ansible_user: "@02858"

Contents of `group_vars` or `host_vars`

ZOAU_HOME:        /hewitt/zopentools/zoau1258
#ZOAU_HOME:        /hewitt/zopentools/zoau1310
#PYZ_HOME:         /hewitt/zopentools/python311/usr/lpp/IBM/cyp/v3r11/pyz
PYZ_HOME:         /hewitt/zopentools/python312/usr/lpp/IBM/cyp/v3r12/pyz

ansible_python_interpreter: "{{PYZ_HOME}}/bin/python3"

environment_vars:
  LIBPATH:        "{{ZOAU_HOME}}/lib:{{PYZ_HOME}}/lib:/lib:/usr/lib:."
  PATH:           "{{ZOAU_HOME}}/bin:{{PYZ_HOME}}/bin:/bin:/usr/sbin:."

  PYTHONSTDINENCODING: cp1047

# not compatible with pip installing the zoau package
# PYTHONPATH:     "{{ZOAU_HOME}}/lib"

# works without it
# LANG: C

  _BPXK_AUTOCVT:  ON
  _CEE_RUNOPTS:   "FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
  _TAG_REDIR_IN:  txt
  _TAG_REDIR_OUT: txt
  _TAG_REDIR_ERR: txt

ketankelkar commented 2 months ago

It looks like you're using the ad-hoc ansible command (docs). That one is tricky to use with z/OS targets because there isn't a clean way to pass in environment variables (which are very helpful in dealing with UTF8/EBCDIC encoding). For this reason, we generally don't recommend the ad-hoc command route. Your playbook above seems alright to me, and if I understood correctly, it was working for you.

(Just an FYI): The recommended environment vars (below) assist with auto-conversion/auto-tagging files. Ansible operates strictly in the UTF-8 encoding space, while z/OS does some of both UTF-8 and EBCDIC.

  _BPXK_AUTOCVT:  ON
  _CEE_RUNOPTS:   "FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
  _TAG_REDIR_IN:  txt
  _TAG_REDIR_OUT: txt
  _TAG_REDIR_ERR: txt

(to the point): I found two ways to enable the command ansible -m ping all. They're both sort of hack-y / undesirable:

First, embed an env command in the ansible python interpreter configuration variable so whenever Ansible attempts invokes the python, it will do so in the environment as defined by the call to env. See zvm1 below:
- contents of inventory.yml:
```
zvm1:
hosts:
    zvm:
        ansible_host: < url_to_host >
        ansible_user: < user_name >
        ansible_python_interpreter: "env PYTHONSTDINENCODING=cp1047 /<python>/3.12/usr/lpp/IBM/cyp/v3r12/pyz/bin/python3"
```
- ad-hoc command and result:
```
ketan@<system> playbooks % ansible -m ping zvm1
zvm | SUCCESS => {
"changed": false,
"ping": "pong"
}
```
  With the PYTHONSTDINENCODING=cp1047 configuration, python tags the stdin pipe and ensures all data is encoded appropriately (and converted to utf-8 when needed). The reason it's important to have the python stdin pipe tagged is so that it is possible to take advantage of the increased speed/performance from enabling pipelining in Ansible (which is enabled by default)
Second, disable pipelining. I've done it via the inventory variable ansible_pipelining below, but it is equivalent to setting the value pipelining = false in the [ssh_connection] section in the ansible.cfg file.
- contents of inventory.yml:
```
zvm2:
hosts:
  zvm:
       ansible_host: < url_to_host >
       ansible_user: < user_name >
       ansible_pipelining: no
       ansible_python_interpreter: "/<python>/3.12/usr/lpp/IBM/cyp/v3r12/pyz/bin/python3"
```
- ad-hoc command and result:
```
ketan@<system> playbooks % ansible -m ping zvm2
zvm | SUCCESS => {
"changed": false,
"ping": "pong"
}
```
  Ansible pipelining, when enabled, leverages the python stdin pipe for I/O, so when data is appropriately tagged (via PYTHONSTDINENCODING environment variable), it gets appropriately converted to the right encoding, and allows for Ansible operations to execute faster since the alternative is for Ansible to copy everything over to temp files (when pipelining is disabled).

You may eventually run into the following error when working with the zos_ping module:

"module_stdout": "< path_to_temp_file >/AnsiballZ_zos_ping.rexx 1: FSUM7332 syntax error: got (, expecting Newline\r\n",

If that happens, set the scp_extra_args parameter like so either in ansible.cfg or the inventory (or group_vars/host_vars).

ansible_scp_extra_args: -O
INI:
```
[ssh_connection]
scp_extra_args = "-O"
```
environment var ANSIBLE_SCP_EXTRA_ARGS: "-O"

Let me know if this helps or if there's anything I didn't answer/address

gngrossi commented 2 months ago

@ketankelkar Success...much appreciated.

bash-4.4$ ansible -m ping all mvs-sysj | SUCCESS => { "changed": false, "ping": "pong" }

I opted for option1... I tried adding to vars: so it could be shared for every host but it doesn't work from there like ansible_user.

For ansible ad-hoc commands, ansible_python_interpreter is defined in the inventory file. For ansible-playbook commands, ansible_python_interpreter is defined in group_vars.

MA1: hosts: mvs-sysj: ansible_python_interpreter: "env PYTHONSTDINENCODING=cp1047 ...." vars: ansible_user: "@02858"

closing...thanks

ansible-collections / ibm_zos_core