New-PASSession using -UseDefaultCredentials getting a hidden error in -ErrorVariable

[AaronG1234]

Describe the issue New-PASSession -UseDefaultCredentials -BaseURI 'https://XXX.YYY.ZZZ' -ErrorVariable Err is Recording an Error into the variable $Err, yet continuing. the module is working, except for the fact my code traps for Errors from that variable.

PS C:\Users\ZZZZZ>  $Err[0].Exception | select *

Status         : ProtocolError
Response       : System.Net.HttpWebResponse
Message        : The remote server returned an error: (500) Internal Server Error.
Data           : {}
InnerException :
TargetSite     : System.Net.WebResponse GetResponse(System.Net.WebRequest)
StackTrace     :    at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(WebRequest request)
                    at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
HelpLink       :
Source         : Microsoft.PowerShell.Commands.Utility
HResult        : -2146233079

---

It is important to note New-PASSession does authenticate, many followup cmdlets do not work ####### Im working on a list to see if I can help narrow the issue##### Get-PASAccount, Get-PASAccountPassword, Set-PASAccount, Unlock-PASAccount, Invoke-PASCPMOperation, do work Get-PASSession, Get-PASAccountActivity 86_5 do not work (The remote server returned an error: (500) Internal Server Error)

Browsers from the same computer work fine (including checking for Account Activity, using Windows Auth from same user, same client computer as the scripts are running) ######################################################

To Reproduce Steps to reproduce the behavior: ######## IMPORTANT, SEE ADDITIONAL CONTEXT #########

1. New-PASSession -UseDefaultCredentials -BaseURI 'https://XXX.YYY.ZZZ' -ErrorVariable Err
2. $Err

   Invoke-WebRequest : The remote server returned an error: (500) Internal Server Error.
   At line:209 char:19
   + ... $APIResponse = Invoke-WebRequest @PSBoundParameters -ErrorAction Stop

   Expected behavior

3. New-PASSession -UseDefaultCredentials -BaseURI 'https://XXX.YYY.ZZZ' -ErrorVariable Err
4. $Err <returns to command prompt because $err is $null>

Screenshots & Console Output If applicable, add screenshots to help explain your problem.

Console Output Code Block:

Your Environment Include relevant details about your environment

• PowerShell Version: 5.1.14393.2879 (also tried 5.1.18362.1474)
• psPAS Version: psPAS 4.8 (and later 5.0)
• CyberArk Version: 11.1.1, PVWA 11.1.1 on IIS (on Server 2019 [1809])

Additional context WHAT CHANGED: Previous to this unexpected result,

• the Application Pool identity was using an Domain User account,
• the SPN of the website users were using: XXX.YYY.ZZZ was set to http/XXX.YYY.ZZZ and was assigned to said user
• but because UseAppPoolCredentia was set to false at system.webServer/security/authentication/windowsAuthentication (for Default Web Site),

therefore Kerberos was failing (since the ticket was made for the domain User, but the AppPool was run in Kernel mode by default and therefore using the computers credentials) and the site was falling back to NTLMv2

Now:

• the Application Pool identity was using an Domain User account,
• the SPN of the website users were using: XXX.YYY.ZZZ was set to http/XXX.YYY.ZZZ and was assigned to said user
• UseAppPoolCredentia was set to true at system.webServer/security/authentication/windowsAuthentication (for Default Web Site)
• Kerberos working as provable by "klist.exe" (searching for http/XXX.YYY.ZZZ in text) run at the client, after klist purge and then using a browser or psPAS to authenticate via Windows Authentication

It is my opinion that the success of the Kerberos is causing an unexpected internal issue in psPAS (but that it is self handling), but the Error from the webrequest is not cleaned up if psPAS does successful continue (so psPAS itself does not 'write-error' or 'return an error' etc., but somewhere in the error stack there was a stop-able error and that error makes it into the cmdlet common parameter -ErrorVariable ____)

[pspete]

Thanks for the detailed report @AaronG1234 , let me study it to see if it is something that will be able to be reproduced - right now, the development environment is not configured for integrated authentication....

[pspete]

Can you see the address in the IIS logs which is related to the 500 error?

Just an idea, does the same issue occur with the following:

New-PASSession -UseDefaultCredentials -BaseURI 'https://XXX.YYY.ZZZ' -SkipVersionCheck -ErrorVariable Err

?

Get-PASAccount/Get-PASAccountPassword - both default to Gen2 API and are working for you Get-PASSession/ Get-PASAccountActivity - both involve Gen1 API /WebServices/PIMServices.svc/ addresses & not working for you.... as the version check uses a Gen1 address, wondering if skipping the version check means the error you see does not get raised.

[AaronG1234]

New-PASSession -UseDefaultCredentials -BaseURI 'https://XXX.YYY.ZZZ' -SkipVersionCheck -ErrorVariable Err

returns no errors (helpful) Get-PASAccountActivity still returns 500 error (this particluar code is using this, but I can stunt this section for a little while)

Do you think that getting to CyberArk 12.1 will improve this?

[AaronG1234]

This is where the 500 Error is

/PasswordVault/WebServices/PIMServices.svc/Accounts/86_5/Activities

[pspete]

Do you think that getting to CyberArk 12.1 will improve this?

Going by the detail you've provided - it seems like only API calls which involve the /PasswordVault/WebServices/PIMServices.svc/ Gen 1 api calls are causing any error to be raised. Calls via Gen 2 /PasswordVault/api/ appear ok?

Be good to figure out why its not getting caught by the module during command invocation, but as some commands work, while others do not, it appears to be a server config issue.

62 & #106 could sound similar

[AaronG1234]

Some Notes on making a PVWA with Windows Auth on PVWA

Setting up a URL that is NOT the hostname of your PVWA https://XXX.YYY.ZZZ/PasswordVault, where the website can be load balanced on multiple IIS servers AND use Windows Authentication:Kerberos (and therefore needs a common name, that is authorized to exactly one Principal in your forest)

---

First: Create/Designate a Domain User or gMSA account to run the Application Pools for PVWA Second: Create/Designate a DNS name that will resolve to your first PVWA server (for troubleshooting) Third: create an Service Principal Name for this IIS based Webserver, setspn -f -s http/fqdn_of_dns_name DOMAIN\User [Or gMSA]

Now proceed with the PVWA Install First Off: SIGN Everything (or make sure your environment doesn't need signed code, CYBERARK SCRIPTS HAVE NO ERROR CHECKING for this failure when the failure is because of Authenticode errors)

#WHERE $Cert = your signing cert, and the current directory is the unzipped PVWA Installation 
Get-ChildItem *.ps1,*.psm1,*.dll,*.exe -Recurse  |  Get-AuthenticodeSignature | ? {​​​​​$_.status -ne "Valid"}​​​​​ | % {​​​​​ Set-AuthenticodeSignature $_.path -cert $cert -IncludeChain notroot -TimestampServer http://timestamp.digicert.com -HashAlgorithm sha256 }​​​​​

After the Hardening and Registration:

1. in Secpol.msc re-add Local Computer\Users to the Local Policy\User Rights Assignment: Access this computer from the network the hardening script is flawed if you are going to use Windows Authentication

==================================================

2. If you are using a GROUP MANAGED SERVICE ACCOUNT, INSTEAD OF JUST A DOMAN USER

   • On another computer with ActiveDirectory module (or adsiedit if you are that raw): 
   • add the computer object to the gMSA's [currently xxxx_PVWA_P] "PrincipalsAllowedToRetrieveManagedPassword" Property in Active Directory (using set-adserviceaccount), do not accidentally remove other servers on the list.

3. Install/validate the gMSA is available for use without using RSAT Tools (Hardening Script removes all unnecessary roles and features) so paste into Powershell from: https://github.com/beatcracker/Powershell-Misc/blob/master/Use-ServiceAccount.ps1

. C:\Users\XXXXX\Desktop\Use-ServiceAccount.ps1
PS C:\Windows\system32> 'xxxx_PVWA_P' | Use-ServiceAccount -Add
PS C:\Windows\system32> 'xxxx_PVWA_P' | Use-ServiceAccount -Query -Detailed

===================================================================

4. add the user you will use for the APPPools to Windows Local\IIS_IUSRS group

5. In IIS while selecting the Default Web site

   • open Management/Configuration Editor
   • system.webServer/security/authentication/windowsAuthentication and set useAppPoolCredentials to True [APPLY]

• open IIS/ASP

• services/Session Properties and set Time-out to 00:41:00 (or 1 minute over whatever your expected timeout is in CyberArk) [APPLY]

• open ASP.NET/Session State

• Cookie Settings/Time-out to 41 (or 1 min over whatever your expected timeout is in CyberArk) [APPLY]

6. in IIS Application Pools

   • set Default Website and PasswordVault Application Pools to
   • run as fullyqualifiedSAMAccount [NBDomain\xxxx_PVWA_P$], and
   • have a idle-time out of 60 (or greater than the .NET time out)

7. in cmd "as Administrator" .... run:

   cd C:\Windows\Microsoft.NET\Framework\v4.0.30319
   .\aspnet_regiis -ga [NBDomain\xxxx_PVWA_P$]
   ::  .\aspnet_regiis -ga fullyqualifiedSAMAccount [NBDomain\SAMAccountname] #:: is a comment in batch

8. while selecting Default WebSite

   • open HTTP Redirect
   • check Redirect Requests to this destination, Redirect All request to exact destination, and Only redirect requests to content in this directory.
   • set destination to: https://FQDNofAlias/PasswordVault/default.aspx

     Make sure DNS or host file points that FQDN to your server, Note: I go all the way to default.aspx because it skips a hidden redirect... Too Many Redirects will cause browsers to halt (they don't like to be clicked around)

9. In the Local Windows Users, Find the CyberArk Scheduling User [PVWAReportsUser], set "Password Never Expires", and save

   • Update the password to a random password now (the old password was LOGGED if you had cmdline logging on),
   • then update the password in the service [CyberArk Scheduled Tasks] Password changes can be done manually or with CPM, but cannot be allowed to expire or bad things happen

10. Add firewall rule allowing 80/443 in Local - Inbound - IIS 80,443 (CyberArk PVWA)

11. Using Notepad (not Notepad++[according to CyberARk]) running as Administrator, open the IIS configuration file. By default, this is %WinDir%\System32\Inetsrv\Config\applicationHost.config  .

At the end of the file, add the following lines:

<location path="Default Web Site/PasswordVault/api/auth/windows/logon">
    <system.webServer>
            <security>
                <authentication>
                    <windowsAuthentication enabled="true" useKernelMode="false">
                        <extendedProtection tokenChecking="None" />
                        <providers>
                            <clear />
                            <add value="Negotiate:Kerberos" />
                        </providers>
                    </windowsAuthentication>
                    <anonymousAuthentication enabled="false" />
                </authentication>
            </security>
        </system.webServer>
</location>

the above is MORE STRICT than CyberArks official Documentation, as it disallows NTLM Fallback, the orignal jsut Enables Windows Auth (which generally starts with NegotiateV1(which has Kerberos and NTLM) and then another line with NTLM,

<location path="Default Web Site/PasswordVault/api/auth/windows/logon">
    <system.webServer>
        <security>
            <authentication>
                    <windowsAuthentication enabled="true" />
            </authentication>
        </security>
    </system.webServer>
</location>

Restart IIS (iisreset) or just reboot

[AaronG1234]

Do you think that getting to CyberArk 12.1 will improve this?

Going by the detail you've provided - it seems like only API calls which involve the /PasswordVault/WebServices/PIMServices.svc/ Gen 1 api calls are causing any error to be raised. Calls via Gen 2 /PasswordVault/api/ appear ok?

Be good to figure out why its not getting caught by the module during command invocation, but as some commands work, while others do not, it appears to be a server config issue.

62 & #106 could sound similar

Yeah, everything headed to /api/... seems happy (as in 200)

[AaronG1234]

P.s. I have to pause on digging in until my production code can finish :), I created a lean version that logged in, and did not read the status/history of the pasaccount. so far only Get-PASSession and the Activity are not working

[AaronG1234]

Ok, definitely server settings... I brought up a new PVWA server and it does not have the issue. too bad I can't pinpoint why (the ApplicationHost.config are practically identical except for encryption for Identity Pool password

[pspete]

Probably Web.config bindings

[pspete]

Fairly confident the error originally reported would be related to this code.

• After the authentication process has completed, Get-PASServer is invoked (Gen1 API) to get the CyberArk version details for use in the version assertion process used throughout the module.
• If Get-PASServer fails, the error is effectively ignored, and version is set to 0.0, meaning future version checks will just pass.

There are numerous closed issues in this project which detail various causes and resolutions for 500 server errors encountered when using the API, the 2 linked to previously may help.

Closing this issue as no immediate changes to be made to the module. An update to Get-PASSession may make its way into a future release so that the session token can be extracted from the module scope to aide troubleshooting without relying on having to get the name of the logging on user (Gen 1 API again).

Feel free to comment if you want the issue reopening or find something which needs any update to the module's code.