AWS authentication fails on a large number of accounts when using credential_process

[jmreicha]

Describe the bug

I am seeing an issue with Steampipe when using a credential manager to "wrap" profiles in encrypted files using credential_process. I don't think it's necessarily an issue with Steampipe but I only see the problem when using Steampipe.

The issue appears when I run a query across a large number of accounts, I usually see it when running across 10+ accounts. The credential_process seems to get overwhelmed taking CPU up to 100% and prompting for a passphrase several times and as a result either fails to cache credentials correctly or has to reload itself.

Steampipe version (steampipe -v)

Steampipe v0.23.1

To reproduce

Example of an aws profile using credential process and sp config, so basically this x60 for the full reproduction.

[profile account1/myrole]
sso_start_url = https://abc123.awsapps.com/start
sso_region = us-west-2
sso_account_name = account1
sso_account_id = 1234567890
sso_role_name = myrole
region = us-west-2
sso_auto_populated = true

[profile sp/account1/myrole]
credential_process = aws-vault exec account1/myrole --json

And the steampipe config.

connection "aws_account1" {
    plugin  = "aws"
    profile = "sp/account1/myrole"
    regions = ["us-*"]
}

Expected behavior

I would expect the credential process to work and not prompt for pin entry.

Additional context

This is on a Linux machine using pass and gpg/gpg-agent for storing the credentials. I have a spare Macbook to test with so will try the configs on it and see what happens since OSX uses the builtin keychain for its credential backend.

The logs don't indicate any issues as most of the entries are unrelated and seem to be normal.

[jmreicha]

Well, I got all the way down into the bottom of the weeds debugging this.

By default, gpg allocates a fixed amount of secure memory. If you don't specify the auto-expand-secmem option the gpg-agent will run out of secure memory if there are a large number of requests it needs to service and restart itself.

Here's the ~/.gnupg/gpg-agent.conf config I have that corrects the issue:

# Cache credentials indefinitely
default-cache-ttl 34560000
max-cache-ttl 34560000
# For debugging purposes
debug-level expert
verbose
log-file /home/jmreicha/.gnupg/gpg-agent.log
# Allow memory to autoexpand
auto-expand-secmem

I figured I would share in case anyone else ever encounters this.