PSWSMan very slow on Mac

jeff-simeon commented 2 years ago

SUMMARY

When running on a Mac, creating a PSSession for ExchangeOnline takes 50 to 75 seconds, whereas the same code, from the same machine, with the same version of PowerShell (7.1.4), from a Windows VM takes 10 to 15 seconds.

LIBMI VERSION

PSWSMan 2.2.1 libcrypto.1.1.dylib

OS / ENVIRONMENT

Mac OS Big Sur

Sample Code

using System;
using System.Diagnostics;
using System.IO;
using System.Management.Automation;
using System.Management.Automation.Runspaces;
using System.Reflection;
using System.Runtime.InteropServices;
using System.Threading.Tasks;

namespace PSWSMan
{
    internal class Program
    {
        private static async Task Main()
        {
            NativeLibrary.SetDllImportResolver(typeof(PSObject).Assembly, ImportResolver);
            await Execute();
        }

        private static async Task Execute()
        {
            var connectionUri =
                "https://outlook.office365.com/powershell-liveid/?BasicAuthToOAuthConversion=true&DelegatedOrg=***";

            var configurationName = "Microsoft.Exchange";

            (string Subject, string AccessToken) token = ("***", "***");

            var sw = Stopwatch.StartNew();
            var rs = RunspaceFactory.CreateRunspace();
            rs.Open();
            using (var ps = PowerShell.Create())
            {
                ps.Runspace = rs;
                var initializationScript = $@"
$global:InformationPreference = 'Continue'
$global:ErrorActionPreference = 'Stop'
$global:ProgressPreference = 'SilentlyContinue'

function Get-Session {{
    Get-PSSession |? ConfigurationName -eq '{configurationName}'
}}

function Remove-Session {{
    foreach($s in Get-Session) {{ 
        try {{ 
            Write-Information ""Removing existing open PSSession $($s.Id) for {configurationName} at {connectionUri}.""        
            $s | Remove-PSSession
        }} 
        catch {{
            Write-Warning ""Encountered an error removing PSSession $($s.Id) ($($_.Exception.Message)).""      
        }}
    }}
}}

function Initialize-Session {{
    $userCredential = New-Object System.Management.Automation.PSCredential('{token.Subject}', (ConvertTo-SecureString 'Bearer {token.AccessToken}' -AsPlainText -Force))
    $option = New-PSSessionOption
    $option.IdleTimeout = [TimeSpan]::FromSeconds(60) # inline setting of this property via New-PSSessionOption is not supported on non-Windows platforms
    $option.OpenTimeout = [TimeSpan]::FromSeconds(60)
    Write-Information 'Creating PSSession for {configurationName} at {connectionUri}.'
    $connectionUri = '{connectionUri}'
    if ($IsMacOS) {{
        # https://github.com/PowerShell/PowerShell/issues/7276
        $connectionUri = $connectionUri.Replace('&', '&amp;')
    }}
    $session = New-PSSession -SessionOption $option -ConfigurationName {configurationName} -ConnectionUri $connectionUri -Credential $userCredential -Authentication Basic -AllowRedirection -WarningAction SilentlyContinue
    $module = Import-PSSession $session -DisableNameChecking -AllowClobber -WarningAction SilentlyContinue
    Import-Module $module -Global -WarningAction SilentlyContinue
    Write-Information ""Imported PSSession $($session.Id) and Module for {configurationName} at {connectionUri} in $($sw.ElapsedMilliseconds)ms.""
}}

try {{ Set-ExecutionPolicy Unrestricted }} catch {{ }} # not supported on non-Windows platforms

if ((Get-Session).State -ne 'Opened') {{
    Remove-Session
    Initialize-Session 
}}";
                await ps.AddScript(initializationScript).InvokeAsync();
            }

            sw.Stop();
            Console.Write(sw.ElapsedMilliseconds);
        }

        public static IntPtr ImportResolver(string libraryName, Assembly assembly, DllImportSearchPath? searchPath)
        {
            if (libraryName == "libpsrpclient")
                libraryName = Path.Combine(Path.GetDirectoryName(typeof(Program).Assembly.Location),
                    $"{libraryName}.dylib");

            return NativeLibrary.Load(libraryName, assembly, searchPath);
        }
    }
}

Any ideas what could be slowing it down?

jborean93 commented 2 years ago

Unfortunately I don't have too much knowledge on this type of performance work but I do have some recommendations for you to try:

Monitor the network traffic with a tool like Wireshark
- Unfortunately for Linux the OMI client doesn't support explicit proxies so you are stuck with just monitoring TLS traffic
- You can at least see whether the slowdown comes from DNS queries, connecting to the hosts, waiting for a response, or somewhere in the client processing the data
Enable OMI logging as per https://github.com/jborean93/omi#troubleshooting
- This gives you more insight into when the client gets the actual data and how long it takes to process it

Most likely this is a problem with the OMI library and there's some performance problem somewhere in the code. Unfortunately this isn't something I've really focused on myself and I wouldn't be surprised if it's a fundamental problem with how the client is set up. If you do find anything interesting I am happy to have a look at it further and see if we can find a solution.

jeff-simeon commented 2 years ago

Thanks @jborean93 - any suggestions on what to look for in the logs?

jeff-simeon commented 2 years ago

@jborean93 - any hints at all would be greatly appreciated

jborean93 commented 2 years ago

The only thing I would look for is to analyze the times it takes to send and receive a response. If it's getting a response in a reasonable time but the session is still slow then something might be up in the code. If it's taking a long time to receive an actual response then it's more likely to be a network problem as OMI is just waiting for a response. Unfortunately this is difficult to fully analyze as it's traffic over HTTPS and everything will be encrypted with Wireshark. The OMI logs should contain a bit more information though.

jeff-simeon commented 2 years ago

I'm not very familiar with these logs so I'm not really sure, but it doesn't seem like there is any specific slowdown over the network. I'm attaching a log file here. If there is anything at all you can see it would be greatly helpful. We would love to switch to Mac for development but this is a blocker for us, as it takes 90 seconds every time we want to start debugging the software.

Thanks for all of your work on this. Really great stuff.

omi-pwsh.log

jborean93 commented 2 years ago

It'll take some time to look into this sorry. If you don't need this for interactive purposes, may I suggest https://github.com/jborean93/pypsrp. It may take a bit of work to implement modern auth but it is definitely possible to run commands on the Exchange PSSession like you would with PowerShell. It's a hell of a lot more stable than the OMI code as well.

jeff-simeon commented 2 years ago

Thanks @jborean93

It's not interactive but we are locked into using PowerShell code for this so I'm not clear on how we'd use the pypsrp library.

jeff-simeon commented 2 years ago

Any new thoughts on this @jborean93 ?

Thanks very much - I really appreciate all of your help with this.

jeff-simeon commented 1 year ago

Hey @jborean93 would you be willing to investigate and try to fix this on a paid basis?

jborean93 / omi