Open HaomouJi opened 1 year ago
Are you able to run - win_whoami
and share the output? I'm curious whether your connection user holds certain privileges. What connection plugin are you using to communicate with the Windows server?
Hi Jordan, thanks for the reply. Pretty sure the cred I put in has local administrator access on remote server. Same cred is working with an ealier version of win_updates module. I will try to revert my fix in the envionrment and add this win_whoami for another test.
I am more curious whether the account has the SeBatchLogonRight
right. This controls whether the newly spawned process is running as the same user or as SYSTEM. I'm unsure why else you would get the access denied error so trying to look at the rights the connection user has might show some more info.
@HaomouJi have you find a solution/workaround for this issue you are having ?
Hi guys, this is Haomou Ji. Sorry for being late, just did the test from working envrionment.
@jborean93 , here is the out come of win_whoami
"rights": [ "SeInteractiveLogonRight", "SeNetworkLogonRight", "SeServiceLogonRight", "SeBatchLogonRight", "SeRemoteInteractiveLogonRight" ], "privileges": { "SeChangeNotifyPrivilege": "enabled-by-default", "SeAssignPrimaryTokenPrivilege": "enabled-by-default", "SeRemoteShutdownPrivilege": "enabled-by-default", "SeBackupPrivilege": "enabled-by-default", "SeLoadDriverPrivilege": "enabled-by-default", "SeSystemProfilePrivilege": "enabled-by-default", "SeShutdownPrivilege": "enabled-by-default", "SeCreatePagefilePrivilege": "enabled-by-default", "SeSystemEnvironmentPrivilege": "enabled-by-default", "SeSystemtimePrivilege": "enabled-by-default", "SeSecurityPrivilege": "enabled-by-default", "SeDelegateSessionUserImpersonatePrivilege": "enabled-by-default", "SeIncreaseWorkingSetPrivilege": "enabled-by-default", "SeTakeOwnershipPrivilege": "enabled-by-default", "SeIncreaseQuotaPrivilege": "enabled-by-default", "SeCreateGlobalPrivilege": "enabled-by-default", "SeCreateSymbolicLinkPrivilege": "enabled-by-default", "SeRestorePrivilege": "enabled-by-default", "SeUndockPrivilege": "enabled-by-default", "SeImpersonatePrivilege": "enabled-by-default", "SeProfileSingleProcessPrivilege": "enabled-by-default", "SeDebugPrivilege": "enabled-by-default", "SeIncreaseBasePriorityPrivilege": "enabled-by-default", "SeManageVolumePrivilege": "enabled-by-default", "SeTimeZonePrivilege": "enabled-by-default" },
@imedaouidene Yes, I got a workaround. I have copied old version of both win_updates.ps1 and win_updates.py files back to the controller host via a task: `- name: WIN PATCHING - PRE-INSTALL - Copy Win_Updates 1.13.0 Module delegate_to: localhost run_once: true ansible.builtin.copy: mode: '755' src: "files/{{ item }}" dest: "/usr/share/ansible/collections/ansible_collections/ansible/windows/plugins/modules/{{ item }}" loop:
Thanks a lot for your workaround @JamesHaomou-Ji-Westpac, it was really helpful. win_updates stopped working for me too after upgrading the collection. Weirdly it is still working for a few servers (especially those that are freshly deployed from template), I can't find the difference between them though. Btw, I had to replace win_updates.py under .../plugins/actions/ too to get it working again, coming from latest version of the collection.
We believe we have narrowed down what part of the code is causing the access is denied error but we haven't figured out why. As I cannot replicate this issue it's very hard to figure out a fix for it. Essentially part of the new win_updates
process is to spawn a new subprocess that does the update work. It is started with the CREATE_NEW_CONSOLE
flag to ensure it isn't tied to the Ansible process which exits shortly after. For whatever reason the presence of the CREATE_NEW_CONSOLE
flag is causing this error.
It would be great if whoever is affected by this could copy the script at https://gist.github.com/jborean93/94b3eea93dbca15e60c51247c3a7b399 to one of the affected servers and then run the following from another Windows host.
Invoke-Command -ComputerName target-host -Command { & "C:\path\where\process_new_console_test.ps1" }
winrs -r:http://target-host:5985/wsman powershell.exe -File "C:\path\where\process_new_console_test.ps1"
The Invoke-Command
example replicates how things run with the psrp
connection plugin while the winrs
command replicates the winrm
connection plugin. It would be good to know if none, one, or both fail in an environment that is affected.
Here the output from your tests:
Invoke-Command -ComputerName test-computer -Command { & "C:\process_new_console_test.ps1" }
Done, exited with rc 1
OperationStopped: Process has exited, so the requested information is not available.
and
winrs -r:http://test-computer:5985/wsman powershell.exe -File "C:\process_new_console_test.ps1"
Exception calling "NativeCreateProcess" with "9" argument(s):
"CreateProcessW() failed (Access is denied, Win32ErrorCode 5 - 0x00000005)"
At C:\process_new_console_test.ps1:644 char:5
+ $procInfo = [Ansible.Windows.Process.ProcessUtil]::NativeCreateProcess(
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], ParentContainsErrorRecordE
xception
+ FullyQualifiedErrorId : Win32Exception
Thanks that at least confirms it's a problem with the way that the winrm
connection plugin works. You could try the win_updates
with the psrp connection plugin. This plugin also uses WinRM as the transport but uses a different shell mechanism on top. It's akin to PowerShell's Invoke-Command
vs the winrs
command from my example.
Now as to why this is happening I'm honestly not sure. It's not something I can replicate or even explain why it might be happening. Knowing any differences between hosts that work might help narrow it down, like:
conhost.exe
in any waywinrs -r:http://target-host:5985/wsman powershell.exe 'gci env: | Sort-object Name'
You could try and use procmon and filter by the winrs process id for testing and see if there's anything that might jump out at you.
I'm not sure if my posting will help at all or confuse things further. Last month I successfully ran win_updates on my servers. Since then I have updated ansible and as a result the win_updates module was updated as well. I am now unable to run win_updates on some servers but on others it still completes successfully. I uninstalled our Anti-virus solution from a server that was failing and now win_updates completed successfully. I asked our security team to verify why the anti-virus solution is blocking the execution of the module. They responded that the issue is most likely not the anti-virus considering nothing changed in the last couple of months on their end and I was provided the following information.
"I’m seeing some differing behavior. When you ran it this morning, I see you connect and create a scheduled task called ‘ansible-ansible.windows.win_updates’. This scheduled task opens cmd.exe, with no parameters. Then you run the task, which opens a command prompt, and then delete the task. All within a few milliseconds of each other. Basically all you’re doing is opening a command prompt. This aligns with the log you posted below in that the task is so short lived, you can’t even get the name of the process running.
I looked further back to when the updates ran on Sept 27th. That time, I see you connect, but instead of creating scheduled tasks, you run an encoded powershell command. I didn’t take the time to decode it, but it’s enough to see that on the 27th, it ran via powershell commands, today, it’s running an empty scheduled task. So, why the difference in behavior? On Sept 27th there’s no scheduled tasks being created, only powershell scripts, but today, it’s creating empty tasks. So, I’m not sure why the change, or what prompted that, but I’d start looking there."
Hope the snippet above adds more clarity on what is occurring in our environment.
@calciosoccer1 thanks for sharing. Essentially the change in behaviour is down to the complexity of this module. What happens is that the Windows Update code needs to run in a context outside of the WinRM network logon as the API doesn't allow this. In the past we used to run the powershell code directly as a scheduled task using -EncodedCommand
or a -File
(depending on your version). This had a bunch of problems around getting the output back reliably or debugging failures in the bootstrapping process that was very hard to diagnose. To counter this we now spawn just a cmd.exe
process that does nothing and use that ne process as the parent of our own powershell process we control. This has a few advantages as now we have a far simpler and secure way to send data to the new powershell process (through stdin) and can easily capture any unreported failures (through stdout/stderr) which we could not do before with having the scheduled task spawn our process. Unfortunately this brings us to the problem in this report where for some reason this is failing and it's not something I can replicate myself.
To try and provide an alternative method we now just spawn a detached process if you are running win_updates
without become removing the scheduled task method altogether but it still uses the parent process spawning step to get working. From what I can gather the problem exists when trying to start with the flag CREATE_NEW_CONSOLE
but I've been unable to find out why it would be failing and even have been unable to replicate the issue on my test hosts so I'm very limited with what I can try out. From what I've seen using the psrp
connection plugin vs the winrm
one can help some people but I've yet to get a confirmation that is actually the case.
I can confirm, it works as expected when I use psrp instead of winrm. Does anyone know if I can use psrp in general instead of winrm?
Yes it should be mostly interchangeable, in fact you would probably notice some slight speed improvements, especially with loops. I can’t say it is 100% the same but they both use WinRM as the underlying transport.
Thank you so much for the quick reply. I managed to switch to psrp and it does work flawlessly. Once again. Thank you so much.
I'm using open source AWX Tower via awx-operator
/Kubernetes and stumbled upon this first suspecting an awx-ee
issue (which is indirectly true, since from 22.2.0
on they're using collections 1.14.0+
.
I do, however, see some integrative issues still there. Trying to switch my project's inventory transport from winrm
to psrp
I found these:
Internal domain servers (Kerberos/ansible_psrp_auth=kerberos
):
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: GSSAPIProxy requires the Python gssapi library: No module named 'gssapi' fatal: [server.domain.tld]: FAILED! => {"changed": false, "failed_update_count": 0, "filtered_updates": {}, "found_update_count": 0, "installed_update_count": 0, "msg": "GSSAPIProxy requires the Python gssapi library: No module named 'gssapi'", "updates": {}}
-> probably just missing gssapi
library in awx-ee
container
DMZ/externally hosted servers (NTLM/ansible_psrp_auth=ntlm
): AWX/Ansible automatically tries to "become" which seems to automatically switch to winrm
as the error resembles the original one (even though ansible_connection=psrp
is set):
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible_collections.ansible.windows.plugins.action.win_updates._ReturnResultException: Failed to start new win_updates task with Ansible Become: Ausnahme beim Aufrufen von "NativeCreateProcess" mit 9 Argument(en): "CreateProcessW() failed (Zugriff verweigert, Win32ErrorCode 5 - 0x00000005)" fatal: [server.domain.tld]: FAILED! => {"changed": false, "failed_update_count": 0, "filtered_updates": {}, "found_update_count": 0, "installed_update_count": 0, "msg": "Failed to start new win_updates task with Ansible Become: Ausnahme beim Aufrufen von \"NativeCreateProcess\" mit 9 Argument(en): \"CreateProcessW() failed (Zugriff verweigert, Win32ErrorCode 5 - 0x00000005)\"", "updates": {}}
SUMMARY
Received error message while using ansible.windows.win_updates. Ansible 1.13.0.
ISSUE TYPE
COMPONENT NAME
ANSIBLE VERSION
COLLECTION VERSION
CONFIGURATION
OS / ENVIRONMENT
windows 2012-2019
STEPS TO REPRODUCE
EXPECTED RESULTS
list available KBs
ACTUAL RESULTS