encoding error on Windows guests with libvirt_qemu connection plugin

[leegarrett]

SUMMARY

When using the qemu guest agent in combination with the community.libvirt.libvirt_qemu connection plugin, all encoding of strings received from a non-english Windows machine will be wrong. They will contain the local encoding (in my example ISO-8859-1), but be treated as utf-8. This will cause problems when trying to use those string in other tasks, or simply outputting it via the debug module.

In ansible < 2.15 this will cause the play to abort. In ansible >= 2.15 this will cause a warning, and the data will be mangled.

This probably didn't show up prominently before, as this problem doesn't occur on Windows machines with english locale, where the character sets (ISO-8559-1 and utf-8) for english letters happen to be identical.

The issue was initially filed against the win_service module, however after some debugging the issue is identified to be within the libvirt connection plugin, as the issue doesn't show up when connecting to the Windows VM via winrm. Details are here: https://github.com/ansible-collections/ansible.windows/issues/546

The error will show up on any win_* module by registering the output, and using the debug module to show the variable. However, for simplicity, a minimal reproducer is provided below.

Likely the libvirt connection module has to detect that it's an windows environment, and change the locale before running the commands, similar to how the ssh connection plugin does: https://github.com/ansible/ansible/blob/bdaa091b33f0ebb273c6ad99b3835530ba2b5a30/lib/ansible/plugins/connection/ssh.py#L1345-L1351

Examples to reproduce shown below.

ISSUE TYPE

• Bug Report

COMPONENT NAME

community.libvirt.libvirt

ANSIBLE VERSION

ansible [core 2.15.3]
  config file = None
  configured module search path = ['/home/randall/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/randall/.local/lib/python3.11/site-packages/ansible
  ansible collection location = /home/randall/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/randall/.local/bin/ansible
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

$ ansible-galaxy collection list community.libvirt

# /home/randall/.local/lib/python3.11/site-packages/ansible_collections
Collection        Version
----------------- -------
community.libvirt 1.2.0  

CONFIGURATION

$ ansible-config dump --only-changed
CONFIG_FILE() = None
EDITOR(env: EDITOR) = /usr/bin/vim
PAGER(env: PAGER) = less

OS / ENVIRONMENT

Debian 12 as ansible controller target node is a German Windows 10 VM running on libvirt, with guest agent installed, connected to via the libvirt connection plugin.

STEPS TO REPRODUCE

1. Install a German Windows 10 VM. Any other non-english language will probably work, too. Windows 11 should work, too.
2. Install guest additions on the guest.

$ cat library/test_encoding.ps1 
#!powershell

#AnsibleRequires -CSharpUtil Ansible.Basic

$module = [Ansible.Basic.AnsibleModule]::Create(@(), @{})
$module.Result.test = "Druckauftr$([char]0xE4)ge"
$module.ExitJson()

$ cat hosts_minimal 
win10work ansible_shell_type=powershell ansible_shell_executable=powershell ansible_connection=community.libvirt.libvirt_qemu

$ ansible -m win_ping -i hosts_minimal win10work
[WARNING]: The "community.libvirt.libvirt_qemu" connection plugin has an improperly configured remote target value, forcing "inventory_hostname" templated value instead of the string
win10work | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

$ ANSIBLE_LIBRARY=./library/ ansible -m test_encoding -i hosts_minimal win10work
[WARNING]: The "community.libvirt.libvirt_qemu" connection plugin has an improperly configured remote target value, forcing "inventory_hostname" templated value instead of the string
[DEPRECATION WARNING]: Module "test_encoding" returned non UTF-8 data in the JSON response. This will become an error in the future. This feature will be removed in version 2.18. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: Non UTF-8 encoded data replaced with "?" while displaying text to stdout/stderr, this is temporary and will become an error. This feature will be removed in version 2.18. Deprecation warnings can be disabled
 by setting deprecation_warnings=False in ansible.cfg.
win10work | SUCCESS => {
    "changed": false,
    "test": "Druckauftr?ge"
}

EXPECTED RESULTS

In the above example it should correctly output "Druckaufträge".

ACTUAL RESULTS

Output from the example is "Druckauftr?ge", or even an error and playbook failure on ansible < 2.15.

[csmart]

Hi @leegarrett, thank you for the bug report and information. I will take a look and see if I can get a patch going to fix it.

[leegarrett]

I've found a solution, I'll send a MR in a bit.