Scaling UDEndpoint

[ChrisMagnuson]

I have an application where I previously ran Envoy proxying requests to 20 separate instances of powershell Polaris to scale the performance of a service we are running.

Using additional instances of polaris we could consume relatively whatever amount of CPU cores we wanted to dedicate to this workload by increasing polaris instances.

I am replacing polaris with UD as it has many advantages but in testing I am not able to get UD to push a system to the limit of the resources it has available and likely due to this it is taking many times longer to complete the work we are feeding it.

I have been reading through UDRunspaceFactory and it seems like there is a notion of a runspace pool where calls to GetRunspace() provide new runspaces that are freed up when the RunspaceReference that contains the runspace is disposed of.

I don't immediately see any obvious limit on the number of runspaces that get created so I would have thought this would scale up to an arbitrarily large size only limited by resource exhaustion, like no more CPU or ram available.

In our current test case we start 20 simultaneous GET requests at a time, is there some configurable limit that would be limiting how many of these request are handled in parallel at given time?

[ChrisMagnuson]

I was mistaken in my testing, there was a delay in a dowstream service that powershell code UD executes was reaching out to which caused UD to be waiting around.

After this has been resolved the system now consumes all the CPU resources available to it.

It still takes roughly double the time that it used to with Polaris so if there are any tips on ways that performance could be increased those would definitely still be appreciated.

[adamdriscoll]

Interesting. I know there are likely things that we can do in UD that will speed this up. One of them is that UD is reparsing the Endpoint every time you execute this. If you have an endpoint with a lot of code, this can cause some slowness. It's certainly possible to correct this. Just need to put in some time to test it.

If you share some snippets of what you are trying to accomplish, I can help with any suggestions. Feel free to email if it's not something you'd like to share with the public.

Otherwise, I'd love to take a stab at speeding up request processing in UD. I think it's something that could benefit everyone using UD.

[ChrisMagnuson]

I have only one endpoint that passes off control to other code which does the heavy lifting for this process so I don't think endpoint parsing would probably be a big deal:

function Start-TervisWebToPrintWebAPIUniversalDashboardRestAPI {
    param (
        [Parameter(Mandatory)]$EndpointInitializationScript
    )
    $Endpoint = New-UDEndpoint -Url "/is/agm/tervis/:TemplateName" -Method "GET" -Endpoint {
        param (
            $TemplateName
        )
        #https://stackoverflow.com/questions/28120222/get-raw-url-from-microsoft-aspnet-http-httprequest 
        $URL = [Microsoft.AspNetCore.Http.Extensions.UriHelper]::GetDisplayUrl($Request)
        $Result = Get-TervisWebToPrintImageFromAdobeScene7WebToPrintURL -RequestURI $URL

        [Microsoft.AspNetCore.Http.HeaderDictionaryExtensions]::Append(
            $Response.Headers,
            "X-InDesignServerPortNumber",
            $Result.SelectedInDesignServerInstancePortNumber
        )

        #Add headers to disable chuncked content-encoding
        [Microsoft.AspNetCore.Http.HeaderDictionaryExtensions]::Append(
            $Response.Headers,
            "Content-Encoding",
            "identity"
        )
        [Microsoft.AspNetCore.Http.HeaderDictionaryExtensions]::Append(
            $Response.Headers,
            "Transfer-Encoding",
            "identity"
        )

        $Response.ContentType = "application/pdf"
        [Microsoft.AspNetCore.Http.SendFileResponseExtensions]::SendFileAsync(
            $Response, 
            $Result.PDFFilePath,
            $(New-Object -TypeName System.Threading.CancellationToken)
        ).GetAwaiter().GetResult()
    }

    $EndpointInitializationScript |
    Set-Content -Path .\InitilizationModule.psm1

    $InitilizationModuleFullName = Get-Item -Path .\InitilizationModule.psm1 |
    Select-Object -ExpandProperty FullName

    $EndpointInitialization = New-UDEndpointInitialization -Module @( $InitilizationModuleFullName )

    Start-UDRestApi -Endpoint $Endpoint -EndpointInitialization $EndpointInitialization -Wait
}

For reference here is the equivalent code used in Polaris:

function Start-TervisWebToPrintPolaris {
    param (
        [Parameter(Mandatory)]$Port
    )
    New-PolarisRoute -Path "/*" -Method "GET" -Scriptblock {        
        $Result = Get-TervisWebToPrintImageFromAdobeScene7WebToPrintURL -RequestURI $Request.url
        $Response.SetContentType("application/pdf")
        $response.SetHeader("X-InDesignServerPortNumber", $Result.SelectedInDesignServerInstancePortNumber)
        $Response.SendBytes($(Get-content -Raw -Encoding Byte -LiteralPath $Result.PDFFilePath))
    } -Force

    $Polaris = Start-Polaris -Https -Port $Port

    while ($Polaris.Listener.IsListening) {
        Wait-Event callbackeventbridge.callbackcomplete
    }
}

In further testing UD is scaling really well for our use case and given that there isn't any performance bottlenecks that you can think of off the top of your head, I don't think this would be worth spending your time on at this point.

[adamdriscoll]

Cool. I did a little test with a completely empty endpoint for both Polaris and UD. Looks like UD is slightly faster just serving requests. The difference it speed could have something to do with the SendFileAsync.

It would be great to take advantage of Async better in UD (or powershell in general) because I think we could get drastically higher throughput that way.

UD

PS C:\Users\adamr> $Endpoint = New-UDEndpoint -Url "/test" -Endpoint { }
PS C:\Users\adamr> Start-UDRestApi -Endpoint $Endpoint -Port 10003

Name        Port Running
----        ---- -------
Dashboard0 10003    True

PS C:\Users\adamr> Measure-Command { 1..1000 | % { Invoke-WebRequest http://localhost:10003/test }  }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 29
Milliseconds      : 253
Ticks             : 292539782
TotalDays         : 0.000338587710648148
TotalHours        : 0.00812610505555555
TotalMinutes      : 0.487566303333333
TotalSeconds      : 29.2539782
TotalMilliseconds : 29253.9782

Polaris

PS C:\Users\adamr> New-PolarisRoute -Path "/*" -Method "GET" -Scriptblock {
>> } -Force
PS C:\Users\adamr> Start-Polaris -Port 10004

App listening on Port: 10004!

Port              : 10004
RouteMiddleWare   : {}
ScriptblockRoutes : {[/*,
                    System.Collections.Generic.Dictionary`2[System.String,System.Management.Automation.ScriptBlock]]}
GetLogsString     : PolarisLogs
ClassDefinitions  :
ContextHandler    : System.AsyncCallback

PS C:\Users\adamr> Measure-Command { 1..1000 | % { Invoke-WebRequest http://localhost:10004/test }  }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 38
Milliseconds      : 0
Ticks             : 380007221
TotalDays         : 0.000439823172453704
TotalHours        : 0.0105557561388889
TotalMinutes      : 0.633345368333333
TotalSeconds      : 38.0007221
TotalMilliseconds : 38000.7221

[adamdriscoll]

Did another (probably more correct test). Running 8 threads (1 per processor)

UD: About 8000 requests in 28 seconds

PS C:\Users\adamr> Measure-Command { 1..8 | % { Start-ThreadJob { 1..1000 | % { Invoke-WebRequest http://localhost:10004
/test  } }  } | Wait-Job }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 28
Milliseconds      : 734
Ticks             : 287344907
TotalDays         : 0.000332575123842593
TotalHours        : 0.00798180297222222
TotalMinutes      : 0.478908178333333
TotalSeconds      : 28.7344907
TotalMilliseconds : 28734.4907

Polaris: 8000 requests in 34 seconds

PS C:\Users\adamr> Measure-Command { 1..8 | % { Start-ThreadJob { 1..1000 | % { Invoke-WebRequest http://localhost:10004
/test  } }  } | Wait-Job }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 34
Milliseconds      : 305
Ticks             : 343053739
TotalDays         : 0.000397052938657407
TotalHours        : 0.00952927052777778
TotalMinutes      : 0.571756231666667
TotalSeconds      : 34.3053739
TotalMilliseconds : 34305.3739

Then I updated UD to use async methods for the web controller API.

Got it down to 22 seconds!

PS C:\Users\adamr> Measure-Command { 1..8 | % { Start-ThreadJob { 1..1000 | % { Invoke-WebRequest http://localhost:10004
/test  } }  } | Wait-Job }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 22
Milliseconds      : 323
Ticks             : 223238770
TotalDays         : 0.000258378206018519
TotalHours        : 0.00620107694444444
TotalMinutes      : 0.372064616666667
TotalSeconds      : 22.323877
TotalMilliseconds : 22323.877

[ChrisMagnuson]

I am working on recreating the results but one thing I noticed is that I got similar times but due to PowerShell's terrible progress bar performance most of time is actually going to that and not requests.

Running your first test of UD with $ProgressPreference = "SilentlyContinue" brought the time down to 8 seconds.

[adamdriscoll]

Ha! Nice!

[LaurentDardenne]

Is it possible to implement for each endpoint a global setting of the variable $ProgressPreference so that it contains the value "SilentlyContinue"?

[adamdriscoll]

The $ProgressPreference is a setting for the client side so setting it within UD wouldn't have any affect.

[LaurentDardenne]

In the code of my endpoints I use RestHeart (The REST API Server for MongoDB), in this case my endpoint is well impacted by $ProgressPreference or am I wrong?

[ChrisMagnuson]

I haven't done an AB test with UD but I saw dramatically better performance for rest api calls made inside of powershell polaris when I included this.

For safe measure I always include this in UD in an EndpointInitializationScript using this technique in UD > 2.0.

UD provides a custom host for its own powershell runspaces and doesn't use the same host that is used when you start a powershell prompt so that may make this uneaded as the progress bar stuff might be host specific.

Would have to test to know for sure.

[adamdriscoll]

Closing this issue as I think we have a resolution. If I'm mistaken, please reopen.