search

jborean93 / pypsrp

PowerShell Remoting Protocol for Python
MIT License
327 stars 49 forks source link

Unable to connect to WSman on WinSrv2019: Server did not respond with a CredSSP token after step Credentials exchange #168

Closed cidrbl0ck closed 1 year ago

cidrbl0ck commented 1 year ago

Environment: RHEL9 AWS AMI Ansible Automation Platform 2.3 Custom Execution Environment: ansible-automation-platform-23/ee-minimal-rhel8:latest built with:

Windows Server 2019 target host:

windowsfirewall: Windows Remote management listening all ip's, allowing all remote sources, direction=in, action=allow WinRM listener: http:5985 Service RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD) MaxConcurrentOperations = 4294967295 MaxConcurrentOperationsPerUser = 1500 EnumerationTimeoutms = 240000 MaxConnections = 300 MaxPacketRetrievalTimeSeconds = 120 AllowUnencrypted = false Auth Basic = false Kerberos = true Negotiate = true Certificate = false CredSSP = true CbtHardeningLevel = Relaxed DefaultPorts HTTP = 5985 HTTPS = 5986 IPv4Filter = IPv6Filter = EnableCompatibilityHttpListener = false EnableCompatibilityHttpsListener = false CertificateThumbprint AllowRemoteAccess = true

PSRemoting enabled WinRM service running "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\LocalAccountTokenFilterPolicy = 1"

Basically all the contents of the standard https://github.com/ansible/ansible/blob/devel/examples/scripts/ConfigureRemotingForAnsible.ps1 have been set except I'm not using a self-signed (or any) certificate and I'm using the HTTP winrm listener versus HTTPS.

Error:

ansible-playbook [core 2.14.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/home/runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.9.13 (main, Nov  9 2022, 13:16:24) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)] (/usr/bin/python3.9)
  jinja version = 3.1.2
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
SSH password: 
setting up inventory plugins
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
redirecting (type: modules) ansible.builtin.win_service to ansible.windows.win_service
Loading collection ansible.windows from /usr/share/ansible/collections/ansible_collections/ansible/windows
redirecting (type: modules) ansible.builtin.win_file to ansible.windows.win_file
redirecting (type: modules) ansible.builtin.win_get_url to ansible.windows.win_get_url
redirecting (type: modules) ansible.builtin.win_package to ansible.windows.win_package
Loading collection community.windows from /usr/share/ansible/collections/ansible_collections/community/windows
redirecting (type: modules) ansible.builtin.win_service to ansible.windows.win_service
redirecting (type: modules) ansible.builtin.win_service to ansible.windows.win_service
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Loading callback plugin awx_display of type stdout, v2.0 from /runner/artifacts/1470/callback/awx_display.py
Attempting to use 'awx_display' callback.
Skipping callback 'awx_display', as we already have a stdout callback.
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: aap-deploy-scaleft-win.yml *******************************************
Positional arguments: aap-standard-win/aap-deploy-scaleft-win.yml
verbosity: 5
remote_user: svc-ansiblemgmt
connection: smart
timeout: 10
ask_pass: True
become: True
become_method: runas
tags: ('all',)
diff: True
inventory: ('/runner/inventory/hosts',)
extra_vars: ('@/runner/env/extravars',)
forks: 5
1 plays in aap-standard-win/aap-deploy-scaleft-win.yml

PLAY [Pre-req installed for Windows Servers] ***********************************

TASK [Gathering Facts] *********************************************************
task path: /runner/project/aap-standard-win/aap-deploy-scaleft-win.yml:4
redirecting (type: modules) ansible.builtin.setup to ansible.windows.setup
Using module file /usr/share/ansible/collections/ansible_collections/ansible/windows/plugins/modules/setup.ps1
Pipelining is enabled.
[WARNING]: ansible_psrp_connection_retries is unsupported by the current psrp
version installed
<10.100.22.79> ESTABLISH PSRP CONNECTION FOR USER: svc-ansiblemgmt ON PORT 5985 TO 10.100.22.79
<10.100.22.79> PSRP OPEN RUNSPACE: auth=credssp configuration=Microsoft.PowerShell endpoint=http://10.100.22.79:5985/wsman
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/usr/lib/python3.9/site-packages/ansible/executor/task_executor.py", line 629, in _execute
    result = self._handler.run(task_vars=vars_copy)
  File "/usr/lib/python3.9/site-packages/ansible/plugins/action/gather_facts.py", line 100, in run
    res = self._execute_module(module_name=fact_module, module_args=mod_args, task_vars=task_vars, wrap_async=False)
  File "/usr/lib/python3.9/site-packages/ansible/plugins/action/__init__.py", line 1167, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python3.9/site-packages/ansible/plugins/action/__init__.py", line 1320, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python3.9/site-packages/ansible/plugins/connection/psrp.py", line 428, in exec_command
    super(Connection, self).exec_command(cmd, in_data=in_data,
  File "/usr/lib/python3.9/site-packages/ansible/plugins/connection/__init__.py", line 35, in wrapped
    self._connect()
  File "/usr/lib/python3.9/site-packages/ansible/plugins/connection/psrp.py", line 392, in _connect
    self.runspace.open()
  File "/usr/local/lib/python3.9/site-packages/pypsrp/powershell.py", line 548, in open
    self.shell.open(options, open_content)
  File "/usr/local/lib/python3.9/site-packages/pypsrp/shell.py", line 205, in open
    response = self.wsman.create(self.resource_uri, shell, option_set=options if len(options.values) else None)
  File "/usr/local/lib/python3.9/site-packages/pypsrp/wsman.py", line 290, in create
    res = self.invoke(WSManAction.CREATE, resource_uri, resource, option_set, selector_set, timeout)
  File "/usr/local/lib/python3.9/site-packages/pypsrp/wsman.py", line 470, in invoke
    response = self.transport.send(xml)
  File "/usr/local/lib/python3.9/site-packages/pypsrp/wsman.py", line 837, in send
    return self._send_request(prep_request)
  File "/usr/local/lib/python3.9/site-packages/pypsrp/wsman.py", line 840, in _send_request
    response = self.session.send(request, timeout=(self.connection_timeout, self.read_timeout))  # type: ignore[union-attr] # This should not happen
  File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 662, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/lib/python3.9/site-packages/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests_credssp/credssp.py", line 199, in response_hook
    response = self.handle_401(response, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests_credssp/credssp.py", line 233, in handle_401
    in_token = self._get_credssp_token(response, credssp_regex, step_name)
  File "/usr/local/lib/python3.9/site-packages/requests_credssp/credssp.py", line 264, in _get_credssp_token
    raise AuthenticationException(error_msg)
requests_credssp.exceptions.AuthenticationException: Server did not response with a CredSSP token after step Credential exchange - actual ''
fatal: [nnnn]: FAILED! => {
    "msg": "Unexpected failure during module execution: Server did not response with a CredSSP token after step Credential exchange - actual ''",
    "stdout": ""
}

PLAY RECAP *********************************************************************
nnnn                       : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

We are migrating away from an AD domain internally to Okta, therefor I'm trying to use a local user account. I've tried both Basic and CredSSP auth. NTLM is old and is mostly disabled on internal systems.

Group Vars (to be perfectly honest I'm not 100% sure this is the appropriate place in AAP): group - os_windows

---
ansible_user: svc-ansiblemgmt
ansible_password: xxxxxxx
ansible_become_method: runas
ansible_become_user: svc-ansiblemgmt
ansible_connection: psrp
ansible_psrp_protocol: http
ansible_psrp_auth: credssp
ansible_psrp_message_encryption: never
ansible_psrp_connection_retries: 3
jborean93 commented 1 year ago

The Credential exchange is the last step of CredSSP and indicates the underlying CredSSP logon with the delegated credentials had failed. Check the security event log on your Windows host to find the login failure message and look at the reason/sub reason for more information.

cidrbl0ck commented 1 year ago

Yeah I've been there Jordan. No errors, successful credential validation, and logon events. In typical Windows fashion, it's no help. But if you are saying it's nothing with the plugin or something I did wrong with the plugin then I will take it and focus efforts elsewhere.

jborean93 commented 1 year ago

The only times I've seen this particular error is when either;

The first scenario cannot happen as the credentials used in the negotiate phase is always going to be the same as the delegation phase so it passing the former means the latter should work.

The second scenario typically happens when the account is a domain account and is not allowed to log on during specific hours or from specific workstations. As this is a local account this shouldn't apply here but there could potentially be other policies at play.

For example here is the Security event log entry when I've manually edited the code to send the wrong credentials during the credential delegation phase

image

It gives me the NTSTATUS of 0xC000006D which is STATUS_LOGON_FAILURE and the sub status of 0xC000006A which is STATUS_WRONG_PASSWORD. Unless these event logs are explicitly disabled I'm not sure why Windows doesn't show it. If that's not the problem and something else is happening you should see at least 3 Logon entries

image

This is the initial negotiate authentication phase

image

image

cidrbl0ck commented 1 year ago

I appreciate it! I'm waiting for a response from the guys who manage the DC's (including GPOs). I suspect there's something under the covers I'm unaware of. Thanks for the help and info!