SSM connection plugin - Windows error

[tremble]

          New version doesn't work.  Pretty sure i've dealt with this error in this PR.  Not 100% which line it was though, and i don't have time right now to figure it out.  This was a windows 2019 instance using AWS's base server ami.  get_facts work, but it doesn't the script is saying it failed.  Hopefully someone else can push this forward, for now i'll just keep using my own copy in our repo.

<i-00bb0b2f67846d1fe> TERMINATE SSM SESSION: AnsibleLambda-08b63421dffecee50
fatal: [i-00bb0b2f67846d1fe]: FAILED! => 
{
    "changed": false,
    "module_stderr": "",
    "module_stdout": "{\"changed\":false,\"invocation\":{\"module_args\":{\"_measure_subset\":false,\"gather_timeout\":10,\"gather_subset\":[\"all\"],\"fact_path\":null}},\"ansible_facts\":{\"ansible_ip_addresses\":[\"fe80::6dcc:b84:27bb:cb96%4\",\"10.216.101.44\"],\"ansible_windows_domain_role\":\"Stand-alone server\",\"ansible_architecture2\":\"x86_64\",\"ansible_user_gecos\":\"\",\"ansible_distribution_major_version\":\"10\",\"ansible_pagefiletotal_mb\":1920,\"ansible_os_name\":\"Microsoft Windows Server 2019 Datacenter\",\"ansible_system_description\":\"\",\"ansible_machine_id\":\"S-1-5-21-2947583606-1977357372-2144821638\",\"ansible_product_serial\":\"ec22cae3-d4fa-\r743e-890a-f3a083a92824\",\"ansible_system_vendor\":\"Xen\",\"gather_subset\":[\"all\"],\"ansible_bios_version\":\"4.2.amazon\",\"ansible_user_id\":\"EC2AMAZ-ML87NO8$\",\"ansible_date_time\":{\"epoch_local\":\"1676980465.82707\",\"epoch_int\":1676998466,\"date\":\"2023-02-21\",\"second\":\"25\",\"tz\":\"Eastern Standard Time\",\"iso8601_micro\":\"2023-02-21T16:54:25.827066Z\",\"iso8601_basic_short\":\"20230221T115425\",\"minute\":\"54\",\"day\":\"21\",\"weekday\":\"Tuesday\",\"iso8601\":\"2023-02-21T16:54:25Z\",\"tz_offset\":\"-05:00\",\"iso8601_basic\":\"20230221T115425827066\",\"epoch\":\"1676998465.82707\",\"weekday_number\":\"2\",\"hour\":\"11\",\"year\":\"2023\",\"month\":\"02\",\"time\":\"11:54:25\",\"weeknumber\":\"7\"},\"ansible_user_dir\":\"C:\\\\Windows\\\\system32\\\\config\\\\systemprofile\",\"ansible_processor_vcpus\":2,\"ansible_user_sid\":\"S-1-5-21-2947583606-1977357372-2144821638-1008\",\"ansible_owner_contact\":\"\",\"ansible_distribution\":\"Microsoft Windows Server 2019 Datacenter\",\"ansible_processor\":[\"0\",\"GenuineIntel\",\"Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz\",\"1\",\"GenuineIntel\",\"Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz\"],\"ansible_powershell_version\":5,\"ansible_uptime_seconds\":4940,\"module_setup\":true,\"ansible_interfaces\":[{\"dns_domain\":\"ec2.internal\",\"connection_name\":\"Ethernet\",\"default_gateway\":\"10.216.101.1\",\"mtu\":9001,\"interface_name\":\"AWS PV Network Device\",\"ipv6\":{\"address\":\"fe80::6dcc:b84:27bb:cb96%4\",\"prefix\":\"64\"},\"macaddress\":\"0E:57:11:F2:2F:CD\",\"speed\":1000,\"interface_index\":4,\"ipv4\":{\"address\":\"10.216.101.44\",\"prefix\":\"25\"}}],\"ansible_distribution_version\":\"10.0.17763.0\",\"ansible_windows_domain_member\":false,\"ansible_swaptotal_mb\":0,\"ansible_hostname\":\"EC2AMAZ-ML87NO8\",\"ansible_virtualization_role\":\"guest\",\"ansible_memfree_mb\":6177,\"ansible_processor_count\":2,\"ansible_os_product_type\":\"server\",\"ansible_netbios_name\":\"EC2AMAZ-ML87NO8\",\"ansible_product_name\":\"HVM domU\",\"ansible_virtualization_type\":\"xen\",\"ansible_os_installation_type\":\"Server\",\"ansible_lastboot\":\"2023-02-21 10:32:06Z\",\"ansible_architecture\":\"64-bit\",\"ansible_bios_date\":\"08/24/2006\",\"ansible_processor_cores\":2,\"ansible_fqdn\":\"EC2AMAZ-ML87NO8\",\"ansible_domain\":\"\",\"ansible_memtotal_mb\":8192,\"ansible_kernel\":\"10.0.17763.0\",\"ansible_pagefilefree_mb\":1709,\"ansible_nodename\":\"EC2AMAZ-ML87NO8\",\"ansible_os_family\":\"Windows\",\"ansible_processor_threads_per_core\":1,\"ansible_owner_name\":\"EC2\",\"ansible_env\":{\"PROCESSOR_IDENTIFIER\":\"Intel64 Family 6 Model 63 Stepping 2, GenuineIntel\",\"ComSpec\":\"C:\\\\Windows\\\\system32\\\\cmd.exe\",\"DriverData\":\"C:\\\\Windows\\\\System32\\\\Drivers\\\\DriverData\",\"LOCALAPPDATA\":\"C:\\\\Windows\\\\system32\\\\config\\\\systemprofile\\\\AppData\\\\Local\",\"Path\":\"C:\\\\Windows\\\\system32;C:\\\\Windows;C:\\\\Windows\\\\System32\\\\Wbem;C:\\\\Windows\\\\System32\\\\WindowsPowerShell\\\\v1.0\\\\;C:\\\\Windows\\\\System32\\\\OpenSSH\\\\;C:\\\\Program Files\\\\Amazon\\\\cfn-bootstrap\\\\;C:\\\\Windows\\\\system32\\\\config\\\\systemprofile\\\\AppData\\\\Local\\\\Microsoft\\\\WindowsApps\",\"SystemRoot\":\"C:\\\\Windows\",\"ALLUSERSPROFILE\":\"C:\\\\ProgramData\",\"CommonProgramFiles(x86)\":\"C:\\\\Program Files (x86)\\\\Common Files\",\"ProgramW6432\":\"C:\\\\Program Files\",\"USERPROFILE\":\"C:\\\\Windows\\\\system32\\\\config\\\\systemprofile\",\"ProgramFiles(x86)\":\"C:\\\\Program Files (x86)\",\"PSModulePath\":\"C:\\\\Program Files\\\\WindowsPowerShell\\\\Modules;C:\\\\Windows\\\\system32\\\\WindowsPowerShell\\\\v1.0\\\\Modules;C:\\\\Program Files (x86)\\\\AWS Tools\\\\PowerShell\",\"PSExecutionPolicyPreference\":\"Unrestricted\",\"USERDOMAIN\":\"WORKGROUP\",\"SystemDrive\":\"C:\",\"EC2LAUNCH_TELEMETRY\":\"1\",\"OS\":\"Windows_NT\",\"CommonProgramFiles\":\"C:\\\\Program Files\\\\Common Files\",\"ProgramFiles\":\"C:\\\\Program Files\",\"USERNAME\":\"EC2AMAZ-ML87NO8$\",\"PROCESSOR_LEVEL\":\"6\",\"COMPUTERNAME\":\"EC2AMAZ-ML87NO8\",\"AWS_EXECUTION_ENV\":\"EC2\",\"NUMBER_OF_PROCESSORS\":\"2\",\"PUBLIC\":\"C:\\\\Users\\\\Public\",\"PROCESSOR_REVISION\":\"3f02\",\"APPDATA\":\"C:\\\\Windows\\\\system32\\\\config\\\\systemprofile\\\\AppData\\\\Roaming\",\"ProgramData\":\"C:\\\\ProgramData\",\"PROCESSOR_ARCHITECTURE\":\"AMD64\",\"CommonProgramW6432\":\"C:\\\\Program Files\\\\Common Files\",\"windir\":\"C:\\\\Windows\",\"PATHEXT\":\".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL\",\"TEMP\":\"C:\\\\Windows\\\\TEMP\",\"TMP\":\"C:\\\\Windows\\\\TEMP\"},\"ansible_windows_domain\":\"WORKGROUP\",\"ansible_reboot_pending\":true,\"ansible_system\":\"Win32NT\"}}",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 0
}

Originally posted by @richardsonky in https://github.com/ansible-collections/community.aws/issues/948#issuecomment-1438823110

[tremble]

@richardsonky - quick question: Which version of the SSM-Agent is installed on your Windows server? There's a comment on https://github.com/ansible-collections/community.aws/issues/1725#issuecomment-1447242011 which seems to imply there are issues connecting to older SSM-Agent versions which may be fixed by updating the SSM-Agent.

[acidtruxs]

@tremble,

We are hitting this bug in our environment when gathering facts. It occurs intermittently but seems to always happen when the chunk of line ends with '\r'. For testing I have a Windows 2019 Server running SSM Agent version: 3.2.1297.0 and we are using the latest aws_ssm plugin from Galaxy.

I believe the issue is in _filter_ansi().

        `line = line.replace("\r\r\n", "\n")`

In the below example "\r\r\n" is spread across multiple lines so the replacement doesn't happen.

Here is a snippet of debug. Output altered to remove identifying info.

EXEC stdout line: 20,"ansible_os_name":"Microsoft Windows Server 2019 Datacenter","ansible_system_description":"","ansible_machine_id":"S-0-0-00-000000000-0000000000-0000000000","ansible_product_serial":"xxxxxxxx-xxxx-\r EXEC stdout line: EXEC stdout line: EXEC stdout line: xxxx-xxxx-xxxxxxxxxxxx","ansible_system_vendor":"Amazon EC2","gather_subset":["all"],"ansible_bios_version":"1.0","ansible_user_id":"CCCCCCC-CCCCCCC$","ansible_date_time":{"epoch_local":"1689342349.07

The first line of output ends with '\r'. The second line is probably the interpreted second '\r'. The third line is probably the interpreted '\n'. The fourth continues the second half of "ansible_product_serial".

For testing I was able to get setup to consistently work by adding an additional regex filter to _filter_ansi() but that broke things in subsequent tasks.

[ansibullbot]

@tremble: Greetings! Thanks for taking the time to open this issue. In order for the community to handle your issue effectively, we need a bit more information.

Here are the items we could not find in your description:

• ansible version
• component name

Please set the description of this issue with this template: https://raw.githubusercontent.com/ansible/ansible/devel/.github/ISSUE_TEMPLATE/bug_report.md

click here for bot help