AuthFailure on long running query

[lobeck]

Describe the bug My config contains ~200 AWS accounts and is configured to query 6 regions each.

I'm now running a query:

select * from aws_vpc_security_group_rule where type = 'ingress' and cidr_ipv4 = '<redacted>/22'

This runs for a while (~15 minutes) when it breaks with the error:

Error: operation error EC2: DescribeSecurityGroups, https response error StatusCode: 401, RequestID: ed9204a7-2950-411b-ab16-1585ebf44ff1, api error AuthFailure: AWS was not able to validate the provided access credentials
operation error EC2: DescribeSecurityGroups, https response error StatusCode: 401, RequestID: 9e2d78d0-715e-434a-9326-143faba67f37, api error AuthFailure: AWS was not able to validate the provided access credentials (SQLSTATE HV000)

I have it running on a Apple M1 using AWS SSO and configured it with your script to generate the config through Organizations.

Steampipe version (steampipe -v) Example: v0.3.0

Plugin version (steampipe plugin list) hub.steampipe.io/plugins/turbot/aws@latest 0.123.0

To reproduce Steps to reproduce the behavior (please include relevant code and/or commands).

Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.

[lobeck]

Did some debugging here.

This is, what CloudTrail sees:

"requestParameters": {
    "roleArn": "arn:aws:iam::XXX:role/XXX",
    "roleSessionName": "steampipe",
    "durationSeconds": 900
},

This is nowhere in my config and nowhere passed explicitly.

So I went searching and apparently, this is the default in aws-sdk-go v1: https://github.com/aws/aws-sdk-go/blob/main/aws/credentials/stscreds/assume_role_provider.go#L127

From the code structure, I can see, that many of the tables still seem to be based on sdk v1 instead of v2

[cbruno10]

Hi @lobeck , sorry you seem to be hitting some timeout errors!

Can you please share some more info on what your connections look like? Are you using an aggregator with multiple accounts? How many regions do you have configured for each account?

Also, if you run a query like:

select * from aws_vpc_security_group_rule where type = 'ingress' and cidr_ipv4 = '<redacted>/22' limit 10

Does Steampipe return results OK?

[lobeck]

No worries, we'll get this sorted 😅

It's an aggregator with ~180 accounts. The config was generated using https://github.com/turbot/steampipe-samples/blob/main/all/aws-organizations-scripts/generate_config_for_cross_account_roles.sh

each account is region limited to:

regions = ["eu-west-1", "eu-central-1", "us-east-1", "us-east-2", "us-west-1", "us-west-2", "ca-central-1"]

The query finishes fine with the added limit statement. I've also run other queries successful like

select instance_id, region, account_id, tags ->> 'Name' as name, metadata_options['HttpTokens'], instance_state from aws_ec2_instance where metadata_options ->> 'HttpTokens' = 'optional';

Even adding

duration_seconds = 3600

to all sections of .aws/config didn't fix it. Also we don't have that many resources in one account, so there must be some overarching session running into the timeout.

[ParthaI]

@cbruno10 @lobeck, I tried the query with SSO credentials for a single account. The query ran for the regions "us-west-2", "us-east-1", "us-west-1", and "us-east-2". But unable to replicate the mentioned error. I'm sharing the observation from my end; please take a look.

I configured my settings following the guidelines in this documentation.

Steampipe config:

connection "aws_sso_aaa" {
  plugin  = "aws"
  profile = "SSO-Admin-aaa"
  regions = ["us-west-2", "us-east-1",  "us-west-1", "us-east-2"]
}

connection "aws_sso_aab" {
  plugin  = "aws"
  profile = "SSO-Admin-aab"
  regions = ["us-west-2", "us-east-1",  "us-west-1", "us-east-2"]
}

connection "aws_all" {
  plugin = "aws"
  type   = "aggregator"
  connections = ["aws_sso_aaa", "aws_sso_aab"]
}

AWS Credential FIle:

[profile SSO-Admin-aab]
sso_session = sso-dev-aab
sso_account_id = xxxxxxxxxxxx
sso_role_name = SSO-Admin
region = us-east-1

[sso-session sso-dev-aab]wx
sso_start_url = https://d-3c672d9842.awsapps.com/start#
sso_region = us-east-2
sso_registration_scopes = xxxxxxxxxxx

[profile SSO-Admin-aaa]
sso_session = sso-dev-aaa
sso_account_id = xxxxxxxxxxxx
sso_role_name = SSO-Admin
region = us-east-1

[sso-session sso-dev-aaa]wx
sso_start_url = https://d-23e3263863.awsapps.com/start#
sso_region = us-east-2
sso_registration_scopes = xxxxxxxxxxx

• I modified the plugin code to adjust the API call.
  • Tested with over 1.6k resources: Duration 63.5s, Rows fetched: 1,609, Hydrate calls: 3,218. (Executed smoothly)
  • With over 3.2k resources: Duration 87.7s, Rows fetched: 3,218, Hydrate calls: 6436. (Executed smoothly)
  • Handling approximately 7k resources: Duration 173.4s, Rows fetched: 4827, Hydrate calls: 9664. (Executed smoothly)
  • For 160k resources, it exceeded an hour. (Resulted in Error)
  • In none of these scenarios was the previously mentioned error encountered. Instead, the API returned a RequestLimitExceeded error (a73f-a580eac163f7, api error RequestLimitExceeded: Request limit exceeded. 2023-11-23 13:18:12.631 UTC [WARN] steampipe-plugin-aws.plugin: [WARN] 1700739268550: BackoffDelay: attempt=1, retryTime=68.25ms, err=https response error StatusCode: 503, RequestID: 668067fb-4d3a-4622-8324-6e9f8a409a1a, api error RequestLimitExceeded: Request limit exceeded.)
• I suspect the error arises due to the high volume of data in the account.
• As per the API documentation, I attempted to reduce the API response by applying filters. However, the API's input parameters have limited support for additional filtering attributes.

@lobeck, I have a few follow-up questions.

• Could you check and inform me if there's an issue with how I've set up the credentials? Additionally, could you provide a guide on how to replicate the following error: Error: operation error EC2: DescribeSecurityGroups, https response error StatusCode: 401, RequestID: ed9204a7-2950-411b-ab16-1585ebf44ff1, api error AuthFailure: AWS was not able to validate the provided access credentials operation error EC2: DescribeSecurityGroups, https response error StatusCode: 401, RequestID: 9e2d78d0-715e-434a-9326-143faba67f37, api error AuthFailure: AWS was not able to validate the provided access credentials (SQLSTATE HV000)?
• Furthermore, try to replicate the same error by adjusting the max_error_retry_attempts and min_error_retry_delay values in the config file, as detailed in these properties.

Thank You!

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[bigdatasourav]

Hey @lobeck, We are closing this issue because we have not heard from you. Please feel free to reopen the issue if you want to share or discuss anything.