search

hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
30.61k stars 4.14k forks source link

Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers) #5878

Closed pawarrchetan closed 4 years ago

pawarrchetan commented 5 years ago

Describe the bug After using the AWS Secrets Backend in VAULT I am getting below error after using the correct credentials stored in VAULT.

[root@ip-100-66-12-90 tf]# vault read aws/creds/my-role

Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

To Reproduce Steps to reproduce the behavior:

  1. vault write aws/config/root \ access_key=ABCDEFGHIJKL \ secret_key=AcVFGCbhjksluyCGD/JHGDPbps2X \ region=eu-central-1

-- Above entry is dummy keys.

vault write aws/roles/my-role \ credential_type=iam_user \ policy_document=-<<EOF
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "iam:DeleteAccessKey", "iam:GetUserPolicy", "iam:GetAccessKeyLastUsed", "iam:UpdateAccessKey", "iam:GetUser", "iam:ListRolePolicies", "iam:CreateAccessKey", "iam:ListAccessKeys" ], "Resource": "arn:aws:iam::80527613***:user/ark-key-rotate" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "iam:GenerateCredentialReport", "iam:GetContextKeysForCustomPolicy", "iam:ListPolicies", "iam:ListUsers", "iam:GetAccountAuthorizationDetails", "iam:GetCredentialReport", "iam:DeleteAccessKey", "iam:GetUserPolicy", "iam:GetAccessKeyLastUsed", "iam:UpdateAccessKey", "iam:GetUser", "iam:ListRolePolicies", "iam:CreateAccessKey", "iam:ListAccessKeys" ], "Resource": "" } ] } EOF Success! Data written to: aws/roles/my-role

vault read aws/creds/my-role Error reading aws/creds/my-role: Get http://127.0.0.1:8200/v1/aws/creds/my-role: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Expected behavior The output should contain the new temporary IAM credentials which can be used to login to AWS CLI.

Environment:

backend "file" {
        path = "/opt/SP/apps/VAULT/data"
}

listener "tcp" {
        tls_disable = 1
}

Additional context I would like to know what could be the problem here. I have also added the credentials for the master user in the .aws/credentials file. but the error still persists.

pawarrchetan commented 5 years ago

In the /var/log/messages file I can see below errors when I try to execute vault read aws/creds/my-role

Nov 30 12:33:04 ip-100-66-12-90 amazon-ssm-agent: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Nov 30 12:34:22 ip-100-66-12-90 dhclient[3289]: XMT: Solicit on eth0, interval 111680ms.
Nov 30 12:35:20 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 INFO [instanceID=i-0eb6e4936f0377681] [HealthCheck] HealthCheck reporting agent health.
Nov 30 12:35:20 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 ERROR [instanceID=i-0eb6e4936f0377681] [HealthCheck] error when calling AWS APIs. error details - NoCredentialProviders: no valid providers in chain. Deprecated.
Nov 30 12:35:20 ip-100-66-12-90 amazon-ssm-agent: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 INFO [instanceID=i-0eb6e4936f0377681] [HealthCheck] increasing error count by 1
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 ERROR [instanceID=i-0eb6e4936f0377681] [HealthCheck] error when calling AWS APIs. error details - NoCredentialProviders: no valid providers in chain. Deprecated.
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Nov 30 12:35:21 ip-100-66-12-90 amazon-ssm-agent: 2018-11-30 12:35:20 INFO [instanceID=i-0eb6e4936f0377681] [HealthCheck] increasing error count by 1
joelthompson commented 5 years ago

Hi @pawarrchetan -- that error most likely means your box doesn't have network connectivity to be able to talk to the AWS IAM endpoints. Does your machine have outbound internet access (e.g., in a subnet with routes to either an IGW or NAT, a security group that permits outbound access to 0.0.0.0/0, no iptables rules on the machine blocking outbound internet access)?

pawarrchetan commented 5 years ago

@joelthompson Thanks for your reply.

however my machine has access to the AWS API endpoints as I can easily use aws cli to communicate with the AWS resources. Please see below : $ nslookup www.google.com Server: 100.66.12.2 Address: 100.66.12.2#53

Non-authoritative answer: Name: www.google.com Address: 172.217.22.4

I can list the S3 buckets in my account using aws cli.

aws s3api list-buckets

{ "Owner": { "ID": "**a115287808afc4a02fb33f4ef4" }, "Buckets": [ { "CreationDate": "2018-09-10T07:59:04.000Z", "Name": "2gj9i-g8s-access-logs" }, { "CreationDate": "2018-11-28T16:12:51.000Z", "Name": "2mpte-g8s-access-logs" },

Regards... Chetan

joelthompson commented 5 years ago

The ListBucket command could be going through an S3 VPC Endpoint, while the IAM calls wouldn't. Can you use the awscli to make an IAM call, e.g., aws iam get-user?

pawarrchetan commented 5 years ago

@joelthompson Yes I can do IAM calls using aws cli

aws iam get-user

{ "User": { "UserName": "terraform-crp", "Path": "/", "CreateDate": "2018-11-08T11:20:01Z", "UserId": "AIDAJJENJGUOS****", "Arn": "arn:aws:iam::80527613****:user/terraform-crp" } }

pawarrchetan commented 5 years ago

@joelthompson

ANy update on the request ?

Regards... Chetan

gauthamatic commented 5 years ago

I'm facing a similar issue but for Assume role credentials. Any update on this?

mdasilva commented 4 years ago

I'm facing the same issue. Is this a confirmed bug?

TomHowarth commented 4 years ago

this is my code sanitized for here.

provider "vault" { address = "${var.vault_addr}" token = "${var.vault_token}" }

resource "vault_aws_secret_backend" "aws" { access_key = "${var.access_key}" secret_key = "${var.secret_key}" region = "us-east-1"

default_lease_ttl_seconds = "120" max_lease_ttl_seconds = "240" }

resource "vault_aws_secret_backend_role" "ec2-admin" { backend = "${vault_aws_secret_backend.aws.path}" name = "my-role" credential_type = "assumed_role"

policy_document=<<EOT { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "", "Resource": "" } ] } EOT }

my vault is running and showing as initialized.

When I run the above code it is successful.

however when I run the following: vault read /aws/creds/my-role I receive the following error

Error reading aws/creds/ec2-admin-role: Error making API request.

URL: GET http://127.0.0.1:8200/v1/aws/creds/my-role Code: 400. Errors:

  • did not supply a role_arn parameter and unable to determine one
spangenberg commented 4 years ago

We have two different issues here.

@TomHowarth your vault_aws_secret_backend is missing the allowed AWS roles to assume, please take a look here: https://www.terraform.io/docs/providers/vault/r/aws_secret_backend_role.html#role_arns

Regarding the first issue from @pawarrchetan, it looks networking related to me. If this still persists, I'd like to know more about the network configuration you're using, how the VPC is set up, which region you're running.

spangenberg commented 4 years ago

We haven't heard back regarding this issue in over 24 days. To try and keep our GitHub issues current, we'll be closing this issue in approximately seven days if we do not hear back regarding this issue. Please let us know if you can still reproduce this issue, and if there is any more information you could share, otherwise we'll be closing this issue.

spangenberg commented 4 years ago

Closing for now. Let us know if you have any more information!