search

ansible-collections / ansible.windows

Windows core collection for Ansible
https://galaxy.ansible.com/ansible/windows
GNU General Public License v3.0
233 stars 157 forks source link

[Windows 11] CS1504 Failed to compile C# code #588

Closed alexschomb closed 4 months ago

alexschomb commented 4 months ago
SUMMARY

Running any Ansible playbook or command against fresh setup Windows 11 hosts fails after a few days / updates. Reproducible issue on currently ~40 identical setup Windows 11 PCs. The issue is likely caused by installing new Windows Malicious Software Removal Tool updates. Windows 10 PCs in the same network/domain are not affected.

ISSUE TYPE
COMPONENT NAME

Any component, even Gathering facts or win_ping is failing.

ANSIBLE VERSION
ansible [core 2.16.3]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
COLLECTION VERSION
Collection                               Version
---------------------------------------- -------
amazon.aws                               7.2.0
ansible.netcommon                        5.3.0
ansible.posix                            1.5.4
ansible.utils                            2.12.0
ansible.windows                          2.2.0
arista.eos                               6.2.2
awx.awx                                  23.6.0
azure.azcollection                       1.19.0
check_point.mgmt                         5.2.2
chocolatey.chocolatey                    1.5.1
cisco.aci                                2.8.0
cisco.asa                                4.0.3
cisco.dnac                               6.10.2
cisco.intersight                         2.0.7
cisco.ios                                5.3.0
cisco.iosxr                              6.1.1
cisco.ise                                2.7.0
cisco.meraki                             2.17.2
cisco.mso                                2.5.0
cisco.nxos                               5.3.0
cisco.ucs                                1.10.0
cloud.common                             2.1.4
cloudscale_ch.cloud                      2.3.1
community.aws                            7.1.0
community.azure                          2.0.0
community.ciscosmb                       1.0.7
community.crypto                         2.17.1
community.digitalocean                   1.26.0
community.dns                            2.8.0
community.docker                         3.7.0
community.general                        8.3.0
community.grafana                        1.7.0
community.hashi_vault                    6.1.0
community.hrobot                         1.9.0
community.library_inventory_filtering_v1 1.0.0
community.libvirt                        1.3.0
community.mongodb                        1.6.3
community.mysql                          3.8.0
community.network                        5.0.2
community.okd                            2.3.0
community.postgresql                     3.3.0
community.proxysql                       1.5.1
community.rabbitmq                       1.2.3
community.routeros                       2.12.0
community.sap                            2.0.0
community.sap_libs                       1.4.2
community.sops                           1.6.7
community.vmware                         4.1.0
community.windows                        2.1.0
community.zabbix                         2.3.1
containers.podman                        1.11.0
cyberark.conjur                          1.2.2
cyberark.pas                             1.0.25
dellemc.enterprise_sonic                 2.4.0
dellemc.openmanage                       8.7.0
dellemc.powerflex                        2.1.0
dellemc.unity                            1.7.1
f5networks.f5_modules                    1.27.1
fortinet.fortimanager                    2.3.1
fortinet.fortios                         2.3.4
frr.frr                                  2.0.2
gluster.gluster                          1.0.2
google.cloud                             1.3.0
grafana.grafana                          2.2.4
hetzner.hcloud                           2.4.1
hpe.nimble                               1.1.4
ibm.qradar                               2.1.0
ibm.spectrum_virtualize                  2.0.0
ibm.storage_virtualize                   2.2.0
infinidat.infinibox                      1.3.12
infoblox.nios_modules                    1.6.1
inspur.ispim                             2.2.0
inspur.sm                                2.3.0
junipernetworks.junos                    5.3.1
kubernetes.core                          2.4.0
lowlydba.sqlserver                       2.2.2
microsoft.ad                             1.4.1
netapp.aws                               21.7.1
netapp.azure                             21.10.1
netapp.cloudmanager                      21.22.1
netapp.elementsw                         21.7.0
netapp.ontap                             22.9.0
netapp.storagegrid                       21.11.1
netapp.um_info                           21.8.1
netapp_eseries.santricity                1.4.0
netbox.netbox                            3.16.0
ngine_io.cloudstack                      2.3.0
ngine_io.exoscale                        1.1.0
openstack.cloud                          2.2.0
openvswitch.openvswitch                  2.1.1
ovirt.ovirt                              3.2.0
purestorage.flasharray                   1.26.0
purestorage.flashblade                   1.15.0
purestorage.fusion                       1.6.0
sensu.sensu_go                           1.14.0
splunk.es                                2.1.2
t_systems_mms.icinga_director            2.0.1
telekom_mms.icinga_director              1.35.0
theforeman.foreman                       3.15.0
vmware.vmware_rest                       2.3.1
vultr.cloud                              1.12.1
vyos.vyos                                4.1.0
wti.remote                               1.0.5
CONFIGURATION
CONFIG_FILE() = /etc/ansible/ansible.cfg
DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = ['/etc/ansible/inventories/staging/hosts.yml', '/etc/ansible/inventories/production/hosts.yml']
DEFAULT_LOAD_CALLBACK_PLUGINS(/etc/ansible/ansible.cfg) = True
DEFAULT_ROLES_PATH(/etc/ansible/ansible.cfg) = ['/etc/ansible/roles/external', '/etc/ansible/roles/local', '/root/.ansible/roles', '/usr/share/ansible/roles', '/etc/ansible/roles']
DEFAULT_STDOUT_CALLBACK(/etc/ansible/ansible.cfg) = community.general.diy
PLAYBOOK_VARS_ROOT(/etc/ansible/ansible.cfg) = all

Note: the DEFAULT_STDOUT_CALLBACK was introduced after the issue appeared and is used for hiding unreachable hosts in the Ansible output.

OS / ENVIRONMENT

Host OS: Debian 12 x64 Target OS: Windows 11 Professional 23H2 x64 Clients are connected to Active Directory / Domain Network. Group Policy Objects are used for setting up CredSSP based authentication for WSMan (confirmed working). AppLocker is not activated. Only Windows Firewall and Windows Defender are in use — no other antivirus or blocking software in use.

Further information can be found in this mailing list discussion (no responses): https://groups.google.com/g/ansible-project/c/jBm3ZSL43TA

STEPS TO REPRODUCE

The minimal test-case is setting up a fresh Windows 11 Pro 23H2 PC (German language). After installation and connection to the AD / Windows Domain the Group Policies set up the WSMan/WinRM connections and all Ansible Playbooks are working just fine. On this day, all currently available Windows Updates (including drivers & firmware) can be installed without causing any issue — even after several reboots. After 3 to 4 days without any other interaction (usually in the new week) suddenly Ansible stops working with this PC. Even shutting off the PC after initial setup with Ansible and cutting power for days will cause the issue on the next boot and installing the latest Windows Updates.

It is unclear which update causes the issue because there are multiple updates available, but it is highly most likely reproduced by the test-case that a previous Windows Malicious Software Removal Tool was automatically installed and after a week a new version/update to this removal tool is installed. The issue with the Ansible connection will happen instantly after update installation, without a reboot necessary. Uninstalling the latest Windows Updates that likely caused the issue doesn't resolve the problem — currently there is no known solution to restore Ansible connections. Deactivation of Windows Firewall and all Windows Defender (including kernel isolation etc.) options doesn't help.

Normal WinRM connections with PowerShell from other Windows PCs are working just fine. Different authentication or connections methods fail with the same error (tried NTLM and OpenSSH). The Ansible Host is not the reason as well, other hosts (Ubuntu WSL2) with much simpler playbooks experience the same error.

Changing the ansible_user (which is a local Windows Admin account, even tested Domain Admin accounts with no success) or deleting/recreating the local Windows profile of this account doesn't help. Allowing Full Permissions to the whole user directory for Everyone doesn't help.

Interestingly, Windows 10 Pro 22H2 x64 hosts in the same domain (affected by the same GPOs) that were executing similar Ansible Playbooks are working just fine, even after months of Windows Updates.

ansible PC123456 -i inventories/production/hosts.yml -m win_ping -vvv
EXPECTED RESULTS

The Ansible Playbooks or commands should be executing as they initially were on the same PC.

ACTUAL RESULTS

Any Ansible connection to the Windows 11 hosts fails immediately with the below error message (partly German) which states that a random filename in the Local AppData TEMP Directory could not be access because of a Unknown Error.

ansible [core 2.16.3]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /etc/ansible/inventories/production/hosts.yml as it did not pass its verify_file() method
script declined parsing /etc/ansible/inventories/production/hosts.yml as it did not pass its verify_file() method
Parsed /etc/ansible/inventories/production/hosts.yml inventory source with yaml plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: __adhoc_playbook__ ****************************************************************************************************************************************************************

PLAY [Ansible Ad-Hoc] ***********************************************************************************************************************************************************************

TASK [win_ping] *****************************************************************************************************************************************************************************
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Using module file /usr/local/lib/python3.11/dist-packages/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<PC123456> ESTABLISH WINRM CONNECTION FOR USER: MYDOMAIN\Admin on PORT 5985 TO PC123456
EXEC (via pipeline wrapper)
The full traceback is:
Failed to compile C# code:
error CS1504: Die Quelldatei 'c:\Users\admin\AppData\Local\Temp\uz3ycsqu.0.cs' konnte nicht geöffnet werden ('Unbekannter Fehler ').
In Zeile:389 Zeichen:13
+             throw [InvalidOperationException]$msg
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], InvalidOperationException
    + FullyQualifiedErrorId : Failed to compile C# code:
error CS1504: Die Quelldatei 'c:\Users\admin\AppData\Local\Temp\uz3ycsqu.0.cs' konnte nicht geöffnet werden ('Unbekannter Fehler ').

ScriptStackTrace:
bei Add-CSharpType, <Keine Datei>: Zeile 389
bei <ScriptBlock>, <Keine Datei>: Zeile 19
bei <ScriptBlock><End>, <Keine Datei>: Zeile 143
bei <ScriptBlock>, <Keine Datei>: Zeile 11
fatal: [PC123456]: FAILED! => {
    "changed": false,
    "msg": "internal error: failed to run exec_wrapper action module_powershell_wrapper: Failed to compile C# code: \r\nerror CS1504: Die Quelldatei 'c:\\Users\\admin\\AppData\\Local\\Temp\\uz3ycsqu.0.cs' konnte nicht geöffnet werden ('Unbekannter Fehler ')."
}

PLAY RECAP **********************************************************************************************************************************************************************************
PC123456                   : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
jborean93 commented 4 months ago

Hi

Thanks for the detailed bug report, it could potentially be that the antivirus policy is removing or blocking access to the temporary files created by Add-CSharpType causing Ansible to fail.

I've created a test VM, installed all the updates available, and I am able to do a win_ping before I joined it to a domain and after without any problems. I'll wait a few days before testing again to see if that changes but unfortunately it's probably more a sign of the AV blocking this action for some reason.

Can you share whether you are using Windows Defender or some other AV product here. Are there any logs indicating it might have blocked access to this temporary path? You could also look into setting a custom ansible_remote_tmp path to something that's excluded by the AV or just not in the user profile dir. You can do this through the ansible_remote_tmp variable for example:

- hosts: win-host
  gather_facts: false
  tasks:
  - win_ping:
    vars:
      ansible_remote_tmp: C:\Windows\TEMP

The connection user must have write access to that directory so you can set it to another one if that's not the case.

alexschomb commented 4 months ago

Hi @jborean93,

Thank you for the prompt response and initial tests! Thanks for your explanation about the Add-CSharpType cmdlet, I was trying to find more information on what is actually happening behind the scenes during an Ansible Windows connection, unsuccessfully. (The next step would have been analyzing the source code.)

Can you share whether you are using Windows Defender or some other AV product here. Are there any logs indicating it might have blocked access to this temporary path?

The affected Windows 11 PCs don't have any other AV product installed, but the default Windows Defender (not the Pro version) delivered with a default Windows 11 install. Unfortunately, Windows Defender doesn't show any detections in the Protection history. We also took a look at the Windows Event Viewer, but couldn't find any relevant errors or warnings during Ansible connection attempts. There are logs concerning the successful login and logout events of the user caused by Ansible.

You could also look into setting a custom ansible_remote_tmp path to something that's excluded by the AV or just not in the user profile dir.

Thank you for the very promising suggestion on ansible_remote_tmp. We will do thorough testing on this tomorrow.

alexschomb commented 4 months ago

I couldn't wait and found a PC to test it with. Unfortunately, setting ansible_remote_tmp didn't help. Will do more thorough testing tomorrow, though.

# ansible PC123456 -i inventories/staging/hosts.yml -m win_ping -e ansible_remote_tmp='C:\Windows\TEMP\' -vvv
ansible [core 2.16.3]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /etc/ansible/inventories/staging/hosts.yml as it did not pass its verify_file() method
script declined parsing /etc/ansible/inventories/staging/hosts.yml as it did not pass its verify_file() method
Parsed /etc/ansible/inventories/staging/hosts.yml inventory source with yaml plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: __adhoc_playbook__ ****************************************************************************************************************************************************************

PLAY [Ansible Ad-Hoc] ***********************************************************************************************************************************************************************

TASK [win_ping] *****************************************************************************************************************************************************************************
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Using module file /usr/local/lib/python3.11/dist-packages/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<PC123456> ESTABLISH WINRM CONNECTION FOR USER: MYDOMAIN\Admin on PORT 5985 TO PC123456
EXEC (via pipeline wrapper)
The full traceback is:
Failed to compile C# code:
error CS1504: Die Quelldatei 'c:\Windows\Temp\5nusa1jp.0.cs' konnte nicht geöffnet werden ('Unbekannter Fehler ').
In Zeile:389 Zeichen:13
+             throw [InvalidOperationException]$msg
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], InvalidOperationException
    + FullyQualifiedErrorId : Failed to compile C# code:
error CS1504: Die Quelldatei 'c:\Windows\Temp\5nusa1jp.0.cs' konnte nicht geöffnet werden ('Unbekannter Fehler ').

ScriptStackTrace:
bei Add-CSharpType, <Keine Datei>: Zeile 389
bei <ScriptBlock>, <Keine Datei>: Zeile 19
bei <ScriptBlock><End>, <Keine Datei>: Zeile 143
bei <ScriptBlock>, <Keine Datei>: Zeile 11
fatal: [PC123456]: FAILED! => {
    "changed": false,
    "msg": "internal error: failed to run exec_wrapper action module_powershell_wrapper: Failed to compile C# code: \r\nerror CS1504: Die Quelldatei 'c:\\Windows\\Temp\\5nusa1jp.0.cs' konnte nicht geöffnet werden ('Unbekannter Fehler ')."
}

PLAY RECAP **********************************************************************************************************************************************************************************
PC123456                   : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
jborean93 commented 4 months ago

Could you potentially try the psrp connection plugin? It runs over WinRM like the winrm connection plugin but at a different layer which might avoid some potential troubles. For a local user you just need to install pypsrp and set the connection vars

ansible_connection: psrp
ansible_port: 5985
ansible_user: MYDOMAIN\Admin
ansible_password: ....

I don't think it will help but it's worth a shot to see if it helps or not.

alexschomb commented 4 months ago

Could you potentially try the psrp connection plugin?

Unfortunately, this doesn't solve the issue as well. Below is the -vvv output:

# ansible PC123456 -i inventories/staging/hosts.yml -m win_ping -vvv
ansible [core 2.16.3]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /etc/ansible/inventories/staging/hosts.yml as it did not pass its verify_file() method
script declined parsing /etc/ansible/inventories/staging/hosts.yml as it did not pass its verify_file() method
Parsed /etc/ansible/inventories/staging/hosts.yml inventory source with yaml plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: __adhoc_playbook__ ***********************************************************************************************************************************************************************************************
PLAY [Ansible Ad-Hoc] ******************************************************************************************************************************************************************************************************
TASK [win_ping] ************************************************************************************************************************************************************************************************************redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Using module file /usr/local/lib/python3.11/dist-packages/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<PC123456> ESTABLISH PSRP CONNECTION FOR USER: MYDOMAIN\Admin ON PORT 5985 TO PC123456
PSRP: EXEC (via pipeline wrapper)
The full traceback is:
Failed to compile C# code: 
error CS1504: Die Quelldatei 'c:\Users\admin\AppData\Local\Temp\jhebpyyq.0.cs' konnte nicht geöffnet werden ('Unbekannter Fehler ').
At line:389 char:13
+             throw [InvalidOperationException]$msg
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], InvalidOperationException
    + FullyQualifiedErrorId : Failed to compile C# code: 
error CS1504: Die Quelldatei 'c:\Users\admin\AppData\Local\Temp\jhebpyyq.0.cs' konnte nicht geöffnet werden ('Unbekannter Fehler ').

ScriptStackTrace:
at Add-CSharpType, <No file>: line 389
at <ScriptBlock>, <No file>: line 19
at <ScriptBlock><End>, <No file>: line 143
at <ScriptBlock>, <No file>: line 11
fatal: [PC123456]: FAILED! => {
    "changed": false,
    "msg": "internal error: failed to run exec_wrapper action module_powershell_wrapper: Failed to compile C# code: \r\nerror CS1504: Die Quelldatei 'c:\\Users\\admin\\AppData\\Local\\Temp\\jhebpyyq.0.cs' konnte nicht geöffnet werden ('Unbekannter Fehler ')."
}

PLAY RECAP *****************************************************************************************************************************************************************************************************************
PC123456                   : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
alexschomb commented 4 months ago

After thorough testing, we can confirm that setting a different path for ansible_remote_tmp does not solve the issue — even for new paths that are accessible for any user will full privileges. Last week, we also set up a Windows 11 PC from a different manufacturer (normal: Lenovo Tiny M70q Gen 4; this time: Intel NUC Gen 8) and cut it from power until today. Just plugged it in again, and while it was working for the first few minutes, suddenly it stopped working and the issue appeared without intervention. From the Windows Update log, we can see that two updates were installed today:

Currently we're looking into the Autounattend.xml file we're using for setting up the PCs. Here is a very similar configuration that we used (we just added other credentials and implemented automatic driver installation during Windows Setup): Schneegans.de Unattend-Generator. While it sounds plausible that the options Harden ACLs or Audit process creation events might cause the issue, we don't understand why the Ansible connection initially works fine for multiple days and then suddenly stops working.

Edit: After comparing the file line by line, it seems that we also had Windows Defender Application Control configured for unrestricted auditing. Apparently, the last command generated by the this option is different now than a few weeks ago. We are using: powershell.exe -NoProfile -File "%TEMP%\wdac.ps1". Auditing only shouldn't block anything though — and the Windows Security options page says that AppLocker is not activated.

alexschomb commented 4 months ago

I'm really sorry. Apparently the generated WDAC policies appear to cause the problem. I found a XML file in C:\Windows\System32\CodeIntegrity\CiPolicies\Active\ (next to several CIP files). Evaluating the file gives the following output:

# Get-CiPoilicy -FilePath "C:\Windows\System32\CodeIntegrity\CiPolicies\Active\{d26bff32-32a2-48a3-b037-10357ee48427}.xml"

Name           : Microsoft Product Root 2010 Windows EKU
Id             : ID_SIGNER_WINDOWS_PRODUCTION_0
TypeId         : Allow
Root           : 06
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : {ID_EKU_WINDOWS}
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : Microsoft Product Root 2010 ELAM EKU
Id             : ID_SIGNER_ELAM_PRODUCTION_0
TypeId         : Allow
Root           : 06
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : {ID_EKU_ELAM}
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : Microsoft Product Root 2010 HAL EKU
Id             : ID_SIGNER_HAL_PRODUCTION_0
TypeId         : Allow
Root           : 06
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : {ID_EKU_HAL_EXT}
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : Microsoft Product Root 2010 WHQL EKU
Id             : ID_SIGNER_WHQL_SHA2_0
TypeId         : Allow
Root           : 06
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : {ID_EKU_WHQL}
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : Microsoft Product Root WHQL EKU SHA1
Id             : ID_SIGNER_WHQL_SHA1_0
TypeId         : Allow
Root           : 05
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : {ID_EKU_WHQL}
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : Microsoft Product Root WHQL EKU MD5
Id             : ID_SIGNER_WHQL_MD5_0
TypeId         : Allow
Root           : 04
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : {ID_EKU_WHQL}
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : Microsoft Flighting Root 2014 Windows EKU
Id             : ID_SIGNER_WINDOWS_FLIGHT_ROOT_0
TypeId         : Allow
Root           : 0E
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : {ID_EKU_WINDOWS}
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : Microsoft Flighting Root 2014 ELAM EKU
Id             : ID_SIGNER_ELAM_FLIGHT_0
TypeId         : Allow
Root           : 0E
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : {ID_EKU_ELAM}
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : Microsoft Flighting Root 2014 HAL EKU
Id             : ID_SIGNER_HAL_FLIGHT_0
TypeId         : Allow
Root           : 0E
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : {ID_EKU_HAL_EXT}
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : Microsoft Flighting Root 2014 WHQL EKU
Id             : ID_SIGNER_WHQL_FLIGHT_SHA2_0
TypeId         : Allow
Root           : 0E
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : {ID_EKU_WHQL}
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : MincryptKnownRootMicrosoftTestRoot2010
Id             : ID_SIGNER_TEST2010_0
TypeId         : Allow
Root           : 0A
FileVersionRef : 
AppIDRef       : 
Wellknown      : True
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : False
attributes     : {}

Name           : C:\Windows\* FileRule
Id             : ID_ALLOW_A_1_1
TypeId         : Allow
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[MinimumFileVersion, 0.0.0.0], [FilePath, C:\Windows\*]}

Name           : C:\Program Files\* FileRule
Id             : ID_ALLOW_A_2_1
TypeId         : Allow
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[MinimumFileVersion, 0.0.0.0], [FilePath, C:\Program Files\*]}

Name           : C:\Program Files (x86)\* FileRule
Id             : ID_ALLOW_A_3_1
TypeId         : Allow
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[MinimumFileVersion, 0.0.0.0], [FilePath, C:\Program Files (x86)\*]}

Name           : C:\Windows\debug\WIA\* FileRule
Id             : ID_DENY_D_4_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\debug\WIA\*]}

Name           : C:\Windows\PLA\Reports\* FileRule
Id             : ID_DENY_D_5_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\PLA\Reports\*]}

Name           : C:\Windows\PLA\Rules\* FileRule
Id             : ID_DENY_D_6_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\PLA\Rules\*]}

Name           : C:\Windows\PLA\Templates\* FileRule
Id             : ID_DENY_D_7_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\PLA\Templates\*]}

Name           : C:\Windows\Registration\CRMLog\* FileRule
Id             : ID_DENY_D_8_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\Registration\CRMLog\*]}

Name           : C:\Windows\System32\Com\dmp\* FileRule
Id             : ID_DENY_D_9_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\System32\Com\dmp\*]}

Name           : C:\Windows\System32\FxsTmp\* FileRule
Id             : ID_DENY_D_A_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\System32\FxsTmp\*]}

Name           : C:\Windows\System32\LogFiles\WMI\* FileRule
Id             : ID_DENY_D_B_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\System32\LogFiles\WMI\*]}

Name           : C:\Windows\System32\Microsoft\Crypto\RSA\MachineKeys\* FileRule
Id             : ID_DENY_D_C_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\System32\Microsoft\Crypto\RSA\MachineKeys\*]}

Name           : C:\Windows\System32\spool\drivers\color\* FileRule
Id             : ID_DENY_D_D_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\System32\spool\drivers\color\*]}

Name           : C:\Windows\System32\spool\PRINTERS\* FileRule
Id             : ID_DENY_D_E_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\System32\spool\PRINTERS\*]}

Name           : C:\Windows\System32\spool\SERVERS\* FileRule
Id             : ID_DENY_D_F_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\System32\spool\SERVERS\*]}

Name           : C:\Windows\System32\Tasks_Migrated\* FileRule
Id             : ID_DENY_D_10_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\System32\Tasks_Migrated\*]}

Name           : C:\Windows\System32\Tasks\* FileRule
Id             : ID_DENY_D_11_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\System32\Tasks\*]}

Name           : C:\Windows\SysWOW64\Com\dmp\* FileRule
Id             : ID_DENY_D_12_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\SysWOW64\Com\dmp\*]}

Name           : C:\Windows\SysWOW64\FxsTmp\* FileRule
Id             : ID_DENY_D_13_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\SysWOW64\FxsTmp\*]}

Name           : C:\Windows\SysWOW64\Tasks\* FileRule
Id             : ID_DENY_D_14_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\SysWOW64\Tasks\*]}

Name           : C:\Windows\Tasks\* FileRule
Id             : ID_DENY_D_15_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\Tasks\*]}

Name           : C:\Windows\Temp\* FileRule
Id             : ID_DENY_D_16_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\Temp\*]}

Name           : C:\Windows\tracing\* FileRule
Id             : ID_DENY_D_17_1
TypeId         : Deny
Root           : 
FileVersionRef : 
AppIDRef       : 
Wellknown      : False
Ekus           : 
Exceptions     : 
FileAttributes : 
FileException  : False
UserMode       : True
attributes     : {[FilePath, C:\Windows\tracing\*]}

Removing the policies solves the issue instantly:

# CiTool.exe -rp "{d26bff32-32a2-48a3-b037-10357ee48427}" -json
{"OperationResult":0}

It is still very misleading that Windows Security says that AppLocker is deactivated, but apparently that is not the case. I guess we will need to write a second policy now that allows Ansible to create files in the local AppData directory. It's also quite confusing that the blocking doesn't happen for the first few days after installation — it feels like AppLocker is being enabled by those Windows Defender updates, although we're having a tough time verifying this. Furthermore, the WDAC policy in place definitely contains a Rule Option Enabled:Audit Mode which means that it shouldn't be blocking anything.

Thanks for your help! I'm sorry that in the end it turned out to be my own error and not a general issue.

alexschomb commented 4 months ago

We finally found a corresponding log entry in the Windows Event Viewer unter Applications and Services Logs > Microsoft > Windows > CodeIntegrity with the following error message:

Code Integrity determined that \Device\HarddiskVolume3\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe is trying to load \Device\HarddiskVolume3\Users\Admin\AppData\Local\Temp\5uetri4n.0.cs which failed the dynamic code trust verification with error code of 0xC0000225.

The detailed view as XML is the following:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-CodeIntegrity" Guid="{4ee76bd8-3cf4-44a0-a0ac-3937643e37a3}" /> 
    <EventID>3114</EventID> 
    <Version>0</Version> 
    <Level>2</Level> 
    <Task>23</Task> 
    <Opcode>134</Opcode> 
    <Keywords>0x8000000000000000</Keywords> 
    <TimeCreated SystemTime="2024-02-27T13:18:44.4677356Z" /> 
    <EventRecordID>1534</EventRecordID> 
    <Correlation /> 
    <Execution ProcessID="4476" ThreadID="1728" /> 
    <Channel>Microsoft-Windows-CodeIntegrity/Operational</Channel> 
    <Computer>PC123456.mydomain.local</Computer> 
    <Security UserID="S-1-5-21-3853710103-2715487670-2987767631-1000" /> 
  </System>
  <EventData>
    <Data Name="FileNameLength">69</Data> 
    <Data Name="FileName">\Device\HarddiskVolume3\Users\Admin\AppData\Local\Temp\5uetri4n.0.cs</Data> 
    <Data Name="ProcessNameLength">76</Data> 
    <Data Name="ProcessName">\Device\HarddiskVolume3\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe</Data> 
    <Data Name="Status">0xc0000225</Data> 
    <Data Name="SHA1HashSize">20</Data> 
    <Data Name="SHA1Hash">57806E8FB39A5CC3E2F0E325C683A18CBAC6F8C2</Data> 
    <Data Name="SHA256HashSize">32</Data> 
    <Data Name="SHA256Hash">3CEAF1887871E12BFC583EB25CC3BF6BB7B76201B27AEE1020E6D2EF5D48104F</Data> 
    <Data Name="SHA256FlatHashSize">32</Data> 
    <Data Name="SHA256FlatHash">3CEAF1887871E12BFC583EB25CC3BF6BB7B76201B27AEE1020E6D2EF5D48104F</Data> 
  </EventData>
</Event>

You don't happen to know how to create a specific policy to allow only such events caused by Ansible? ChatGPT suggest the following solution (if feasible for Ansible?):

Certificate Rules: If csc.exe and its output are signed with a trusted certificate, you could create a certificate rule that trusts anything signed by that certificate. This is a more secure approach, as it ensures only code signed by a trusted entity is allowed to run.

By the way, looking at the logs of Applications and Services Logs > Microsoft > Windows > AppLocker indicates that AppLocker is indeed working only in Audit mode and not blocking anything (as it is supposed to do).

Looking at the documentation of Microsoft for WDAC policy options, I suppose that this is the reason Code Integrity is still blocking Ansible:

19 Enabled:Dynamic Code Security Enables policy enforcement for .NET applications and dynamically loaded libraries.NOTE: This option is only supported on Windows 10, version 1803 and later, or Windows Server 2019 and later.NOTE: This option is always enforced if any WDAC UMCI policy enables it. There's no audit mode for .NET dynamic code security hardening.

jborean93 commented 4 months ago

Awesome I am glad you figured out what policy is causing the problem, really great investigation work there. Unfortunately I'm not sure of any way to just allow Ansible to perform this work as what you allow for Ansible would really just apply to anything that does a network logon through winrs.vbs or Invoke-Command/Enter-PSSession on PowerShell.

Unfortunately the fundamental nature of how Ansible works means it needs to be able to run whatever the module wants. We don't really have a good story around this. I know this might not be the news you wanted to hear but we need to be able to compile the code using Add-Type on a network logon to perform all these operations sorry.

alexschomb commented 4 months ago

Thank you for your assistance! I'm very relieved to have found a cause and solution to the issue after weeks of debugging.

I understand that the fundamental nature of Ansible conflicts with the Dynamic Code Security policies of AppLocker. I think the only solution to this might be signed & trusted files/processes, but I understand that this is way beyond the scope of an Open-Source product.

I hope that this ticket might help others experiencing similar or same error messages with Ansible in the future, as there was nearly no information (and certainly no clear answer) to this issue on the internet. I'd like to close this GitHub issue with the solution we're likely to implement now.

I did some testing on how to add supplemental policies for AppLocker that would allow Ansible processes that are blocked by Dynamic Code Security. I found that neither File Hash rules nor File Path or Name rules happen to exclude certain processes from DCS. I don't understand everything from this very detailed and interesting article, but my interpretation is that the security researcher / author confirms my suspicion. As a result, the only way to allow Ansible access to the system with ANY AppLocker policy in place (no matter if Audit only or Enforced) is to remove the Enabled:Dynamic Code Security section from the XML file and recompile to binary (CIB file).

A simple way to do this (without much manual work) is using a WDAC Policy Generator like WDAC Wizard or Schneegans.de WDAC-Policy-Generator. The latter doesn't offer an option to disable Dynamic Code Security, so you'll need to manually remove that section and recompile to binary OR just edit the XML with WDAC Wizard and disable DCS in the Advanced Options, save, implement the new files and reboot. That way, your system might be a target to dynamic code attacks, but at least you can block other known attacks, for example by implementing the regularly updated recommended Microsoft deny rules — easy options for such are available in both tools.

jborean93 commented 4 months ago

Thank you very much for your investigation work, it will definitely help others who might be on the same track as you are. Sorry this wasn't the answer you are looking for but appreciate the time you've taken to look into this.