PowerShell / SecretManagement

PowerShell module to consistent usage of secrets through different extension vaults
MIT License
335 stars 47 forks source link

Long delay in getting 1rst secret / unlocking vault in powershell core on Linux #176

Closed eoq closed 3 years ago

eoq commented 3 years ago

When running powershell core (7.1.4 or 7.2.0) on Linux (both RHEL7 and RHEL8) it takes somewhere between a 4-8 seconds to get the 1rst secret from a SecretStore local vault and then < 10 ms per request after that. In powershell core (7.1.3) on Windows it only takes ~350 ms for the 1rst request and < 20 ms per request after.

That long delay is problematic for some otherwise short running scripts.

What could be the cause of this? How best to troubleshoot further?

Thanks Ed

Testing method:

$PSVersionTable ; "nTESTING:" ; 1..4 | % {(Measure-Command {Get-Secret -Name testsecret -AsPlainText}).TotalMilliseconds}`

Sample results:

> $PSVersionTable ; "`nTESTING:" ; 1..4 | % {(Measure-Command {Get-Secret -Name testsecret -AsPlainText}).TotalMilliseconds}

Name                           Value
----                           -----
PSVersion                      7.1.4
PSEdition                      Core
GitCommitId                    7.1.4
OS                             Linux 3.10.0-1160.31.1.el7.x86_64 #1 SMP Wed May 26 20:18:08 UTC 2021
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

TESTING:
6197.1455
10.5208
16.4169
15.6045
PaulHigin commented 3 years ago

It is difficult to know what is causing the initial delay. It could be CLR JITing or maybe file system caching. I have not seen or heard of anything this severe on any platform. Are you running on a resource limited system?

iSazonov commented 3 years ago

You could try to explicitly load secret modules and see if the delay is at the load time.

eoq commented 3 years ago

I think I found the issue though I can't explain it. I did a system trace and noticed that when I start powershell and also when I run get-secret, for some reason something is doing an fstat call on all of our filesytems of all types. We use autofs automounter and this triggers all the NFS mounts to happen while it waits.

Any idea how I could track this down to the source in powershell or .NET? Do you know of a way to profile a module so I can find the call that is triggering this?

Thanks Ed

iSazonov commented 3 years ago

I think it is known issue https://github.com/PowerShell/PowerShell/issues/15061

eoq commented 3 years ago

Thanks for finding that. I will follow-up over there.