HTTPS request timeouts when Pode is running on PS 7.4

[jfromh]

Describe the Bug

I'm having an issue with Invoke-WebRequest/Invoke-RestMethod timeouts using SSL when Pode is running on PowerShell 7.4 but not 7.2. This is similar to #977 but this seems to be a server-side issue. On the client side, I have tried 7.2 and 7.4 but it doesn't seem to matter, and the browser doesn't seem to be affected by this.

Steps To Reproduce

Server:

Start-PodeServer {
    New-PodeLoggingMethod | Enable-PodeErrorLogging -Level 'Error', 'Debug', 'Verbose'

    Add-PodeEndpoint -Name 'HTTPS' -Address '*' -Protocol 'HTTPS' -Port 443 -CertificateStoreName 'My' -CertificateStoreLocation 'LocalMachine' -CertificateThumbprint $Thumbprint

    Add-PodeRoute -Method 'GET' -Path '/test' -ScriptBlock {
        Write-PodeTextResponse -Value (Get-Date)
    }
}

Client:

While ($True) {
    (Invoke-WebRequest -Method 'GET' -Uri $URI -TimeoutSec 1).Content; Start-Sleep -Seconds 1
}

Expected Behavior

When Pode is running on 7.2 I get an updated timestamp every second. When Pode is running on 7.4 I get mostly timeouts with random successful timestamps in between.

Platform

• OS: Windows
• Browser: Chrome
• Versions:
  • Pode: 2.10.0
  • PowerShell: 7.4.2

[mdaneri]

It's not a Pode issue but an Invoke-WebRequest one try with curl.exe, and will work Check this pester code https://github.com/Badgerati/Pode/blob/develop/tests/integration/RestApi.Https.Tests.ps1 we have exactly the same issue

[jfromh]

I've come across the Invoke-WebRequest issues in other projects with PS5.1 but this problem only shows up when the Pode server is running on PS7.4. When running on PS7.2 I get no timeouts with Invoke-WebRequest.

[G2ft]

Hello, I've the same issue. My server is behind a haproxy (https://myserver.mydomain.com redirect to https://myserver.com:9443). Live, it works correctly in 7.3.2, but when I upgrade to PowerShell 7.4 it doesn't work anymore. The first request arrives correctly and then loops indefinitely until a timeout occurs. Without going through the HAproxy, https://myserver.com:9443 it works with version 7.4. A collateral effect of the .net update to version 8? I don't know if this will be useful. Have a nice day.

[phdavis]

We were able to replicate this issue easily after some further troubleshooting. It appears that if we use HTTPS on 7.4.x with Pode, API calls with IRM or IWR are about 85% successful, while the remainder timeout for no obvious reasons. We can tell in our logs that the routes appear to be completing and returning results, but the JSON seemingly never returns, so our invokes time out.

This randomness and timeouts cause things to run extremely slow, to the point we cannot use 7.4.x for our Pode Server. If we downgrade to 7.3.12 everything works as expected without changing anything.

Broken

Add-PodeEndpoint -Hostname "$APIHostname" -Address * -Port 443 -Protocol Https -Name HTTPSV1 -CertificateThumbprint "$HTTPCertThumbprint" -CertificateStoreLocation 'LocalMachine'

Works

Add-PodeEndpoint -Hostname "$APIHostname" -Address * -Port 80 -Protocol Http -Name HTTPV1'

Server Pode 2.10.1 PS 7.4.4 Windows 2022 servers TLS1.2 or TLS1.3 30 Threads

Client Anything (?). Tested with PS5 and PS7 clients. Web clients (Chrome) behave the same.

We get no errors, warnings, or exceptions anywhere. Here's an example test looping through one of our routes. We have all log types enabled (Informational, Error, Warning, Debug, Verbose). It sounds like https://github.com/Badgerati/Pode/issues/977 may be the same issue, but perhaps not realized.

Note this can be any endpoint, even root /
$Test = Invoke-RestMethod "https://servername.tld.com/api/test/contacts?ServerName=testserver" -Headers $Headers -Method Get -TimeoutSec 5

Oddly enough if we switch to Curl as mentioned by @mdaneri above, everything works.

$result = (curl -s -X GET "https://servername.tld.com/api/test/contacts?ServerName=testserver" -k -H 'Authorization: Bearer XYZ123' ) | ConvertFrom-Json

Additionally we've tried adding an IPv6 setup, as well as -DualMode which doesn't appear to make a difference. The same behavior was mentioned prior to the commit of DualMode in https://github.com/Badgerati/Pode/pull/1267

[phdavis]

Testing on PS7.4.5 today appears to be still be problematic.

[Badgerati]

I've just pushed a refactor of the .NET socket receiver which Pode uses to the branch/commit above.

I was able to reproduce the issue described, and with the work done above was able to resolve the issue on my end. If there's any one in a possible to checkout the branch above, build and test Pode locally, that would be a great help!

[mdaneri]

I tested it, and it works fine on the Windows VM, where it always fails. On Linux an Mac a never seen this issue Please remember to remove the curl test from the RestApi.Https.Tests.ps1. If you can keep both curl and invoke-RestMethod, it will be great

[G2ft]

Test in Windows server 2022 PS 7.4.4 and 7.4.5 and it's ok for me. Thanks for your investigation :)

[phdavis]

Working fine here now on Server 2022, the test build, and PS7.4.5. FWIW I can't tell a noticeable difference in performance, but that's hard to gauge in our dev environment since it's pretty quiet.

[Badgerati]

Awesome, thanks all! 😄

I'll get this merged for 2.11.0 then