aap_callback failing to setup WinRM today

UNiXMIT commented 11 months ago

Summary

Yesterday (11th July 2023) my Ansible scripts were working OK. They use aap_callback to enable WinRM. The my script waits for port 5986 to become available using ansible.builtin.wait_for. Worked fine for months. Today (approx. lunchtime) it stopped working and my EC2 instances are no longer working with WinRM on port 5986. After my ansible script fails I try to telnet 5986 and sure enough it's not open. I can't understand what's changed.

Issue Type

Bug Report

Component Name

ec2_instance

Ansible Version

ansible [core 2.13.3]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/support/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /home/support/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.9.14 (main, Jan  9 2023, 00:00:00) [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)]
  jinja version =

Collection Versions

Collection                    Version
----------------------------- -------
amazon.aws                    5.4.0  
ansible.netcommon             3.1.0  
ansible.posix                 1.4.0  
ansible.utils                 2.6.1  
ansible.windows               1.11.0 
arista.eos                    5.0.1  
awx.awx                       21.4.0 
azure.azcollection            1.13.0 
check_point.mgmt              2.3.0  
chocolatey.chocolatey         1.3.0  
cisco.aci                     2.2.0  
cisco.asa                     3.1.0  
cisco.dnac                    6.5.3  
cisco.intersight              1.0.19 
cisco.ios                     3.3.0  
cisco.iosxr                   3.3.0  
cisco.ise                     2.5.0  
cisco.meraki                  2.10.1 
cisco.mso                     2.0.0  
cisco.nso                     1.0.3  
cisco.nxos                    3.1.0  
cisco.ucs                     1.8.0  
cloud.common                  2.1.2  
cloudscale_ch.cloud           2.2.2  
community.aws                 3.5.0  
community.azure               1.1.0  
community.ciscosmb            1.0.5  
community.crypto              2.5.0  
community.digitalocean        1.21.0 
community.dns                 2.3.1  
community.docker              2.7.1  
community.fortios             1.0.0  
community.general             5.5.0  
community.google              1.0.0  
community.grafana             1.5.2  
community.hashi_vault         3.2.0  
community.hrobot              1.5.2  
community.libvirt             1.2.0  
community.mongodb             1.4.2  
community.mysql               3.4.0  
community.network             4.0.1  
community.okd                 2.2.0  
community.postgresql          2.2.0  
community.proxysql            1.4.0  
community.rabbitmq            1.2.2  
community.routeros            2.2.1  
community.sap                 1.0.0  
community.sap_libs            1.2.0  
community.skydive             1.0.0  
community.sops                1.3.0  
community.vmware              2.8.0  
community.windows             1.11.0 
community.zabbix              1.8.0  
containers.podman             1.9.4  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.14 
dellemc.enterprise_sonic      1.1.1  
dellemc.openmanage            5.5.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.19.0 
fortinet.fortimanager         2.1.5  
fortinet.fortios              2.1.7  
frr.frr                       2.0.0  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.8.1  
hpe.nimble                    1.1.4  
ibm.qradar                    2.0.0  
ibm.spectrum_virtualize       1.9.0  
infinidat.infinibox           1.3.3  
infoblox.nios_modules         1.3.0  
inspur.sm                     2.0.0  
junipernetworks.junos         3.1.0  
kubernetes.core               2.3.2  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.19.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.22.0
netapp.storagegrid            21.10.0
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.3.1  
netbox.netbox                 3.7.1  
ngine_io.cloudstack           2.2.4  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.2  
openstack.cloud               1.8.0  
openvswitch.openvswitch       2.1.0  
ovirt.ovirt                   2.2.3  
purestorage.flasharray        1.13.0 
purestorage.flashblade        1.9.0  
purestorage.fusion            1.0.2  
sensu.sensu_go                1.13.1 
servicenow.servicenow         1.0.6  
splunk.es                     2.0.0  
t_systems_mms.icinga_director 1.31.0 
theforeman.foreman            3.4.0  
vmware.vmware_rest            2.2.0  
vyos.vyos                     3.0.1  
wti.remote                    1.0.4

AWS SDK versions

WARNING: Package(s) not found: boto
Name: boto3
Version: 1.26.118
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires: jmespath, s3transfer, botocore
Required-by: pfsso
---
Name: botocore
Version: 1.29.118
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires: urllib3, jmespath, python-dateutil
Required-by: s3transfer, boto3

Configuration

No output shown

OS / Environment

cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="9.0 (Plow)" ID="rhel" ID_LIKE="fedora" VERSION_ID="9.0" PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux 9.0 (Plow)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/9/" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.0 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.0"

Steps to Reproduce

- name: SSO 2FA
  hosts: local
  gather_facts: no
  tasks:
  - name: Import sso.yml
    import_tasks: sso.yml
  no_log: True
  tags: default

- name: AWS EC2 Management
  hosts: local
  vars:
    tempName: SEM-{{ 2048 | random }}
  gather_facts: no
  tasks:
  - name: Create AWS EC2 instance
    amazon.aws.ec2_instance:
      aap_callback:
        windows: true
        set_password: '{{ myPassword }}'
      name: '{{ tempName }}'
      image_id: '{{ awsAMI }}'
      key_name: '{{ keyName }}'
      network:  
        assign_public_ip: yes
      security_groups: '{{ securityGroups }}'
      region: '{{ awsRegion }}'
      availability_zone: '{{ availabilityZone }}'
      aws_profile: '{{ awsProfile }}'
      instance_type: '{{ instanceType }}'
      count: '{{ vmCount | default(1) }}'
      volumes:
      - device_name: '{{ deviceName }}'
        ebs:
         volume_type: '{{ volumeType }}'
         volume_size: '{{ volumeSize }}'
      state: running
      wait: true
    register: ec2
    tags: default

  - name: Wait 30 seconds for public IP and AAP Setup
    pause:
      seconds: 30
    tags: default

  - name: Add Host to awsEC2 Group 
    add_host: 
      hostname: '{{ item.public_ip_address }}'
      ansible_user: '{{ ansibleUser }}'
      ansible_password: '{{ myPassword }}'
      ansible_connection: winrm
      ansible_winrm_transport: basic
      ansible_winrm_server_cert_validation: ignore
      ansible_port: 5986
      ansible_winrm_operation_timeout_sec: 120
      ansible_winrm_read_timeout_sec: 140
      groups: awsEC2
    loop: '{{ ec2.instances }}'
    tags: default

  - name: Rename AWS EC2 instance
    amazon.aws.ec2_tag:
      aws_profile: '{{ awsProfile }}'
      region: '{{ awsRegion }}'
      resource: '{{ item }}'
      tags:
        Name: '{{ semaphore_vars.task_details.username|upper }}-{{ imageName }}'
        Owner: '{{ ssoEmail }}'
        OS: '{{ imageName }}'
      state: present
    loop: '{{ ec2.instance_ids }}'
    tags: default

  - name: WinRM Check
    ansible.builtin.wait_for:
      host: '{{ item.public_ip_address }}'
      port: 5986
      timeout: 120
    loop: '{{ ec2.instances }}'
    tags: default

Expected Results

Previously, before today, 'ansible.builtin.wait_for' would wait for port 5986 to become available and then the play would continue on successfully once it was available.

Actual Results

Timeout when waiting for x.x.x.x:5986

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansibullbot commented 11 months ago

Files identified in the description:

[plugins/modules/ec2_instance.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/ec2_instance.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot commented 11 months ago

cc @jillr @ryansb @s-hertel @tremble click here for bot help

UNiXMIT commented 11 months ago

I've checked the EC2 instance that was created and it seems that it's only listening on port 5985 now. Previously it worked on port 5986. Has the WinRM setup script changed in the past day?

tremble commented 11 months ago

Yup, something changed upstream on the 11th.... https://github.com/ansible/ansible/pull/81011

tremble commented 11 months ago

@UNiXMIT If you have a support contract with Red Hat for Ansible Automation Platform I'd strongly recommend opening a case there to get the attention of the Product folks. As part of amazon.aws this would be covered by your support contracts.

The trouble is that the script URL that was hard coded (https://github.com/ansible-collections/amazon.aws/blob/main/plugins/module_utils/tower.py) now points to a file which has been deleted...

UNiXMIT commented 11 months ago

@tremble Hopefully they will change the URL to point to the new one - https://github.com/ansible/ansible-documentation/blob/devel/examples/scripts/ConfigureRemotingForAnsible.ps1 I'm not using AAP. I use aap_callback to set the admin password and enable WinRM.

UNiXMIT commented 11 months ago

I've got it working again. I ditched aap_callback and now use user_data:

user_data: |
    <powershell>
    net user administrator {{ myPassword }}
    iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/ansible/ansible-documentation/devel/examples/scrConfigureRemotingForAnsible.ps1'))
    </powershell>

tremble commented 11 months ago

@UNiXMIT,

Glad you've got something working. With the password in the user_data I'd strongly recommend using no_log: True on that task to avoid accidentally logging your Admin passwords in plain text. https://docs.ansible.com/ansible/latest/reference_appendices/faq.html#keep-secret-data

I've poked a couple of folks behind the scenes as I'm not sure pointing to ansible-docs is the right solution either.

UNiXMIT commented 11 months ago

@tremble Yeah good point. I have no_log set elsewhere so I'll use it here too. Thanks

ansible-collections / amazon.aws