Get-IAMUser is painfully slow

chscott commented 3 weeks ago

Describe the bug

Not much more to add. It's really, really slow. This is on a machine with ample resources that runs other AWS commands without a problem.

Measure-Command { Get-IAMUser }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 16
Milliseconds      : 313
Ticks             : 163138747
TotalDays         : 0.000188817994212963
TotalHours        : 0.00453163186111111
TotalMinutes      : 0.271897911666667
TotalSeconds      : 16.3138747
TotalMilliseconds : 16313.8747

Expected Behavior

I would expect it to run much quicker, though maybe I don't understand the cmdlet fully.

Current Behavior

As described above

Reproduction Steps

Install 4.1.636
Run the command shown above

Possible Solution

No response

Additional Information/Context

No response

AWS Tools for PowerShell version used

AWS.Tools.IdentityManagement 4.1.636

PowerShell version used

Name                           Value
----                           -----
PSVersion                      7.4.4
PSEdition                      Core
GitCommitId                    7.4.4
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Operating System and version

Windows 11

bhoradc commented 3 weeks ago

Hello @chscott,

Thank you for reporting the issue. I tried reproducing the slowness in execution of Measure-Command { Get-IAMUser }, but couldn't replicate it.

$PsVersionTable

Name                           Value
----                           -----
PSVersion                      7.4.4
PSEdition                      Core
GitCommitId                    7.4.4
OS                             Microsoft Windows 10.0.19045
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Binary     4.1.636               AWS.Tools.Common                    Core,Desk {Clear-AWSHistory, Set-AWSHistoryConfig…
Binary     4.1.636               AWS.Tools.IdentityManagement        Core,Desk {Add-IAMClientIDToOpenIDConnectProvider…

Result of Measure-Command { Get-IAMUser }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 75
Ticks             : 751803
TotalDays         : 8.70142361111111E-07
TotalHours        : 2.08834166666667E-05
TotalMinutes      : 0.001253005
TotalSeconds      : 0.0751803
TotalMilliseconds : 75.1803

The performance of the Get-IAMUser cmdlet could be impacted by the number of IAM users in your AWS account. Do you have a large number of IAM users?
If not, can you please enable logging to see if it helps in identifying the slowness. PowerShell logging can be enabled with below commands.
```
Add-AWSLoggingListener -Name "AWS Logs" -LogFilePath "/ps-logs.txt" # or any path
Set-AWSResponseLogging Always
Enable-AWSMetricsLogging
```
Kindly share the logs with us for further review(after redacting sensitive information).

Regards, Chaitanya

chscott commented 3 weeks ago

@bhoradc We have less than 10 IAM users. The output is below. It appears to be the same issue as https://github.com/aws/aws-tools-for-powershell/issues/330.

Amazon Information: 0 : The environment variable AWS_ENABLE_ENDPOINT_DISCOVERY was not set with a value.
Amazon Information: 1 : The environment variable AWS_MAX_ATTEMPTS was not set with a value.
Amazon Information: 2 : The environment variable AWS_RETRY_MODE was not set with a value.
Amazon Information: 3 : The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT was not set with a value.
Amazon Information: 4 : The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE was not set with a value.
Amazon Information: 5 : The environment variable AWS_EC2_METADATA_V1_DISABLED was not set with a value.
Amazon Information: 6 : The environment variable AWS_USE_DUALSTACK_ENDPOINT was not set with a value.
Amazon Information: 7 : The environment variable AWS_USE_FIPS_ENDPOINT was not set with a value.
Amazon Information: 8 : The environment variable AWS_IGNORE_CONFIGURED_ENDPOINT_URLS was not set with a value.
Amazon Information: 9 : The environment variable AWS_DISABLE_REQUEST_COMPRESSION was not set with a value.
Amazon Information: 10 : The environment variable AWS_REQUEST_MIN_COMPRESSION_SIZE_BYTES was not set with a value.
Amazon Information: 11 : The environment variable AWS_SDK_UA_APP_ID was not set with a value.
Amazon Information: 0 : There is no defaults_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 1 : There is no endpoint_discovery_enabled set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 2 : There is no retry_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 3 : There is no max_attempts set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 4 : There is no ec2_metadata_service_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 5 : There is no ec2_metadata_service_endpoint_mode set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 6 : There is no ec2_metadata_v1_disabled set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 7 : There is no use_dualstack_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 8 : There is no use_fips_endpoint set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 9 : There is no ignore_configured_endpoint_urls set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 10 : There is no endpoint_url set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 11 : There is no disable_request_compression set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 12 : There is no request_min_compression_size_bytes set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Information: 13 : There is no sdk_ua_app_id set in the profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
Amazon Error: 0 : Unable to contact EC2 Metadata service to obtain a metadata token. Attempting to access IMDS without a token., System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 5 seconds elapsing.
 ---> System.TimeoutException: A task was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Amazon.Runtime.Internal.Util.AsyncHelpers.<>c__DisplayClass1_0`1.<<RunSync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Amazon.Runtime.Internal.Util.AsyncHelpers.ExclusiveSynchronizationContext.BeginMessageLoop()
   at Amazon.Runtime.Internal.Util.AsyncHelpers.RunSync[T](Func`1 workItem)
   at Amazon.Util.AWSSDKUtils.ExecuteHttpRequest(Uri uri, String requestType, String content, TimeSpan timeout, IWebProxy proxy, IDictionary`2 headers)
   at Amazon.Util.EC2InstanceMetadata.FetchApiToken(Int32 tries)
Amazon Error: 1 : Unable to contact EC2 Metadata service., System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 5 seconds elapsing.
 ---> System.TimeoutException: A task was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Amazon.Runtime.Internal.Util.AsyncHelpers.<>c__DisplayClass1_0`1.<<RunSync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Amazon.Runtime.Internal.Util.AsyncHelpers.ExclusiveSynchronizationContext.BeginMessageLoop()
   at Amazon.Runtime.Internal.Util.AsyncHelpers.RunSync[T](Func`1 workItem)
   at Amazon.Util.AWSSDKUtils.ExecuteHttpRequest(Uri uri, String requestType, String content, TimeSpan timeout, IWebProxy proxy, IDictionary`2 headers)
   at Amazon.Util.EC2InstanceMetadata.GetItems(String relativeOrAbsolutePath, Int32 tries, Boolean slurp, String token)
Amazon Information: 0 : Resolved DefaultConfigurationMode for RegionEndpoint [us-east-1] to [Legacy].
Amazon Verbose: 0 : Received response (truncated to 1024 bytes): [<GetUserResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
  <GetUserResult>
    <User>
      REDACTED BUT VALID
    </User>
  </GetUserResult>
  <ResponseMetadata>
    <RequestId>95a0fc80-0a6f-40a4-8c3f-bc8e1ff62b29</RequestId>
  </ResponseMetadata>
</GetUserResponse>
]
Amazon Information: 1 : Request metrics: AsyncCall = True; CanonicalRequest = POST\n/\n\ncontent-type:application/x-www-form-urlencoded; charset=utf-8\nhost:iam.amazonaws.com\nuser-agent:AWSPowerShell.Common/4.1.636.0 ua/2.0 os/windows#10.0.22631.0 md/ARCH#X64 lang/.NET_Core#8.0.7 md/aws-sdk-dotnet-core#3.7.400.6 api/IAM#3.7.402.0 md/PowerShellCore/7.-1 cfg/retry-mode#legacy md/ClientAsync cfg/init-coll#1\nx-amz-content-sha256:63cdc6da6b76dfdbbd82fe53059afd80e08926740ab5436bd2d4196da1af20db\nx-amz-date:20240817T140027Z\n\ncontent-type;host;user-agent;x-amz-content-sha256;x-amz-date\n63cdc6da6b76dfdbbd82fe53059afd80e08926740ab5436bd2d4196da1af20db; StringToSign = AWS4-HMAC-SHA256\n20240817T140027Z\n20240817/us-east-1/iam/aws4_request\na0bba8c02d420186526fb63180f8270bafa3b81ef5350aa619d21c840004d0d7; ServiceName = Amazon.IdentityManagement; ServiceEndpoint = https://iam.amazonaws.com/; MethodName = GetUserRequest; RequestSize = 33; StatusCode = OK; BytesProcessed = 533; AWSRequestID = 95a0fc80-0a6f-40a4-8c3f-bc8e1ff62b29; CredentialsRequestTime = 00:00:00.0005808; RequestSigningTime = 00:00:00.0091964; HttpRequestTime = 00:00:00.7107565; ResponseUnmarshallTime = 00:00:00.0041333; ResponseProcessingTime = 00:00:00.0060931; ClientExecuteTime = 00:00:00.7493908;

C:\Users\Chad\Downloads> Measure-Command { Get-IAMUser }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 32
Milliseconds      : 567
Ticks             : 325679176
TotalDays         : 0.000376943490740741
TotalHours        : 0.00904664377777778
TotalMinutes      : 0.542798626666667
TotalSeconds      : 32.5679176
TotalMilliseconds : 32567.9176

C:\Users\Chad\Downloads> Measure-Command { Get-IAMUser -Region 'us-west-1' }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 584
Ticks             : 5844861
TotalDays         : 6.76488541666667E-06
TotalHours        : 0.00016235725
TotalMinutes      : 0.009741435
TotalSeconds      : 0.5844861
TotalMilliseconds : 584.4861

chscott commented 3 weeks ago

bhoradc commented 2 weeks ago

Hi @chscott,

Thank you for providing the logs. I am still unable to reproduce this issue at my end, even when I remove the region from credentials file. Can you kindly help answering below queries?

Is the default region set through any configuration, when you execute Measure-Command { Get-IAMUser }?
Per the logs you provided, there's an IMDS connection timing out after 5 seconds. Are you intending to provide credentials via Amazon EC2 instance metadata?
I believe you are running this command locally, and not inside an EC2 machine, is that right?

Regards, Chaitanya

chscott commented 2 weeks ago

I'm not aware of default region being set by other means, but I'm not an expert with this module.
I don't know about providing credentials via Amazon EC2 instance metadata, so I'll say no, that's not my intention. I'm working under the assumption that the credentials are being picked up automatically from my credentials file in the $HOME/.aws directory.
Correct. I'm running the command locally.

normj commented 2 weeks ago

@chscott What I understand is you have no region configured for your PowerShell session. Whether that is through the AWS_REGION environment variable, in profile configuration in $HOME/.aws directory or setting the region on the profile. Without a region configured the PowerShell via the .NET SDK will search for credentials through all of the possible ways which includes EC2 metadata. The search in EC2 metadata is what is taking a long time to fail because it has some retry logic.

In most services after EC2 metadata if the region wasn't detected the cmdlet would have failed with no region configured. IAM is a bit special in that it is considered a "global" service. So the SDK defaults to us-east-1 as the global endpoint when all other region resolutions fail. In hindsight I wish we wouldn't have had the fallback to us-east-1 but at this point I can't remove it because I don't know who that would break.

In your case you should treat the slow call for Get-IAMUser as an error that region is not set and decided how you want to set it for your environment.

github-actions[bot] commented 2 weeks ago

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

aws / aws-tools-for-powershell