AWS environment variables are not honored

[electriquo]

Describe the bug AWS CLI and AWS SDK honor AWS environment variables, while steampipe does not and displays the error

ERROR: rpc error: code = Unknown desc = operation error Elastic Load Balancing v2: DescribeLoadBalancers, https response error StatusCode: 403, RequestID: 9c80b7d8-6cc7-4d20-ae5d-7955fcde037a, api error AccessDenied: User: arn:aws:sts::1234:assumed-role/foo/i-xxx is not authorized to perform: elasticloadbalancing:DescribeLoadBalancers because no identity-based policy allows the elasticloadbalancing:DescribeLoadBalancers action (SQLSTATE HV000)

Steampipe version (steampipe -v)

$ steampipe --version
Steampipe v0.20.2

Plugin version (steampipe plugin list)

$ steampipe plugin list
+---------------------------------------------+---------+-------------+
| Installed Plugin                            | Version | Connections |
+---------------------------------------------+---------+-------------+
| hub.steampipe.io/plugins/turbot/aws@0.103.0 | 0.103.0 | aws         |
+---------------------------------------------+---------+-------------+

To reproduce

1. Expose AWS environment variables

   $ env | grep -o '^AWS.*='
   AWS_DEFAULT_REGION=
   AWS_REGION=
   AWS_ROLE_ARN=
   AWS_WEB_IDENTITY_TOKEN_FILE=
   AWS_STS_REGIONAL_ENDPOINTS=

2. Verify AWS environment variables are working, has ReadOnlyAccess role policy and there are no AWS CLI credential file

   $ ls -x1R ~/.aws/
   /root/.aws/:
   cli
   
   /root/.aws/cli:
   cache
   
   $ aws --version
   aws-cli/2.11.21 Python/3.11.3 Linux/5.4.204-113.362.amzn2.x86_64 exe/x86_64.debian.11 prompt/off
   
   $ aws iam list-attached-role-policies --role-name $(aws sts get-caller-identity --query Arn --output text | rev | cut -d / -f 2 | rev)
   {
      "AttachedPolicies": [
          {
              "PolicyName": "ReadOnlyAccess",
              "PolicyArn": "arn:aws:iam::aws:policy/ReadOnlyAccess"
          }
      ]
   }

3. Install steampipe aws plugin
4. Start steampipe dashboard
5. Open one of the AWS dashboards, e.g. /aws_insights.dashboard.ec2_application_load_balancer_detail

Expected behavior AWS plugin should honor AWS environment variables.

Additional context

• AWS environment variables docs for AWS_ROLE_ARN states:

  If defined, this environment variable overrides the value for the profile setting role_arn. You can't specify a role session name as a command line parameter.

• https://hub.steampipe.io/plugins/turbot/aws#configuring-aws-credentials

If there is no bug, then any guidance on how to make AWS plugin to honor AWS environment variables will be appreciated.

[cbruno10]

Hi @foolioo , can you please let us know which AWS environment variables specifically aren't working for you? Is it just AWS_ROLE_ARN, or are there others as well?

Are you running from an EC2 instance, or another environment, like a local laptop?

Also, if you run steampipe query using the same AWS Steampipe connections you're using with that load balance detail page, are you able to run queries like select name from aws_iam_role, select instance_id from aws_ec2_instance, and select name from aws_s3_bucket?

[electriquo]

@cbruno10

an you please let us know which AWS environment variables specifically aren't working for you? Is it just AWS_ROLE_ARN, or are there others as well?

I cannot know exactly which environment variable isn't honored by steampipe even when trying to set STEAMPIPLE_LOG_LEVEL to trace. Could you specify explicitly how to get the information that you need?

Are you running from an EC2 instance, or another environment, like a local laptop?

Kindly clarify why this should matter when AWS environment are available for steampipe?

Also, if you run steampipe query using the same AWS Steampipe connections you're using with that load balance detail page, are you able to run queries like select name from aws_iam_role, select instance_id from aws_ec2_instance, and select name from aws_s3_bucket?

I use the steampipe dashboard and that is where the error appears and same as before, it is unclear why it matters but here you go :)

psql (15.3, server 14.2)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, compression: off)
Type "help" for help.

steampipe=> select name from aws_iam_role limit 1;
ERROR:  rpc error: code = Unknown desc = operation error IAM: ListRoles, https response error StatusCode: 403, RequestID: cc000269-3591-4c0d-bdd6-d08bf25b0baf, api error AccessDenied: User: arn:aws:sts::1234:assumed-role/foo/i-xxx is not authorized to perform: iam:ListRoles on resource: arn:aws:iam::1234:role/ because no identity-based policy allows the iam:ListRoles action

I can confirm that not authorized to perform: iam:ListRoles is incorrect since ReadOnlyAccess policy (as shown above) was granted and here is an expended policy list of it that includes IAMReadOnlyAccess policy

$ aws iam list-policies --scope AWS --no-only-attached --query 'Policies[].PolicyName' --no-cli-pager | grep IAMReadOnlyAccess
    "IAMReadOnlyAccess",

[electriquo]

@cbruno10 Maybe https://github.com/aws/aws-sdk-go-v2/issues/2867 is the corresponding issue and seems like https://github.com/hasheddan/stack-aws/blob/5680d65ec37d585a3fc5c8f8112d3680b38cccfd/pkg/clients/aws.go#L123-L145 is the solution (https://github.com/aws/aws-sdk-go-v2/issues/475#issuecomment-582956641)

[cbruno10]

@foolioo Can you please share the following information:

• What's in your ~/.steampipe/config/aws.spc (with sensitive info removed)?
• What's in ~/.aws/credentials and ~/.aws/config (with sensitive info removed again)?
• What are the exact steampipe commands you're running when using dashboards or queries?

For the environment variables you listed in your original comment, AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE, I haven't used these before, so seeing how you use them with your AWS credentials and config file would be helpful for us to reproduce the error from our end.

And thanks for sharing those issues. It looks like support for the web identity credential provider was added in AWS SDK Go v2 v0.20.0, and we currently v1.44.189. So I'm not sure if we need the workaround code you linked in the plugin, or if there's something shorter we can add if our plugin is not compatible with that authentication method today.

[electriquo]

@cbruno10

What's in your ~/.steampipe/config/aws.spc?

connection "aws" {
  plugin = "aws"
  regions = ["*"]
  profile = "default"
}

What's in ~/.aws/credentials and ~/.aws/config

Both files do not exist. Otherwise I wouldn't report that AWS environment variables are not honored

What are the exact steampipe commands you're running when using dashboards or queries?

steampipe service start --foreground --dashboard --dashboard-listen=network

[cbruno10]

Thanks @foolioo for the additional info, we'll look to try and reproduce from our end using the environment variables you listed above and let you know what we find.

[electriquo]

@cbruno10 Where you able to reproduce and find the issue?

[cbruno10]

Hi @foolioo , no not yet, we're still investigating. We'll share more info once we have it.

[graza-io]

May not resolve issue but could you maybe try setting the env var AWS_SDK_LOAD_CONFIG=true

I did attempt a basic example of testing environment variables for authentication (access/secret key combination) & that worked out fine.

❯ source .tempenv
❯ ls -x1R ~/.aws/
❯ ll -a ~/.aws
total 0
drwxr-xr-x   2 graza  staff    64B 31 May 10:28 .
drwxr-x---+ 42 graza  staff   1.3K 31 May 10:46 ..
❯ aws --version
aws-cli/2.11.23 Python/3.11.3 Darwin/22.5.0 exe/x86_64 prompt/off
❯ env | grep -o '^AWS.*='
AWS_SDK_LOAD_CONFIG=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=
❯ steampipe query "select count(*) from aws_s3_bucket"
+-------+
| count |
+-------+
| 121   |
+-------+

Also would it be possible to test if writing the env vars to relevant config/credentials files works successfully with Steampipe?

[e-gineer]

@foolioo I think you should remove this from your spc file:

profile = "default"

since you don’t have any aws config files defined then I assume you do not have a default profile to reference?

[graza-io]

Good catch, I didn't set profile in my aws.spc

❯ cat ~/.steampipe/config/aws.spc
connection "aws" {
  plugin = "aws"
  regions = ["eu-*"]
}

When adding the line back in I do indeed get a (different) but understandable error.

❯ vi ~/.steampipe/config/aws.spc
❯ cat ~/.steampipe/config/aws.spc
connection "aws" {
  plugin = "aws"
  regions = ["eu-*"]
  profile = "default"
}

❯ steampipe query "select count(*) from aws_s3_bucket"

Error: operation error S3: ListBuckets, exceeded maximum number of attempts, 9, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, exceeded maximum number of attempts, 3, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: no route to host (SQLSTATE HV000)

+-------+
| count |
+-------+
+-------+

[graza-io]

Hey @foolioo - can you check/confirm that removing the profile from your .spc file allowed you to successfully use the environment variables as expected?

[electriquo]

@graza-io Confirmed. Removing the profile seems to resolve the issue. Will conduct a deeper testing later on and will reopen the issue if it persists