search

hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
31.31k stars 4.23k forks source link

Vault Error: context deadline exceeded on usage of IAM boundary with path_role attribute in IAM role #13442

Open T0tt1 opened 2 years ago

T0tt1 commented 2 years ago

Describe the bug Using server Vault v1.4.2. I am creating a role using type aws/iam. If I create a IAM role which do have role_path property and instance_profile_path property defined in Terraform, once I create in vault the role and make the boundary to the afore mentioned IAM role, I am unable to login in and getting the error Error: context deadline exceeded It works with arn:aws:iam::XXXXXXXX:role/staging-github-action-runners-runner-role-toad and getting the mentioned error while ARN looks like arn:aws:iam::XXXXXXXXXXX:role/hashicorp-manager/staging-github-action-runners-runner-role-toad

To Reproduce Steps to reproduce the behavior:

  1. Create via TF IAM role which produces following arn (set in TF role_path and instance_profile_path) ==> arn:aws:iam::XXXXXXXXXXX:role/hashicorp-manager/staging-github-action-runners-runner-role-toad
  2. Run vault auth enable aws
  3. Run vault write auth/aws/config/client sts_endpoint=https://sts.us-east-1.amazonaws.com sts_region=us-east-1
  4. Run `cat << EOF | vault policy write master-policy -

    Root

    path "*" {
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

    EOF`

  5. Run vault write \ auth/aws/role/master-role \ auth_type=iam \ policies=master-policy \ resolve_aws_unique_ids=false \ max_ttl=1h \ token_ttl=1h \ bound_iam_principal_arn=${runner_instance_profile_role}
  6. SSH on EC2 instance where IAM role is assigned
  7. Run vault login -method=aws role=master-role
  8. See error ==> Error authenticating: context deadline exceeded

Expected behavior No matter of the fact whether I do have role_path and instance_profile_path used while creating IAM role, once I provide the arn during the creation of role in Vault I must be able to login. Creating an IAM role with arn arn:aws:iam::XXXXXXXXXXX:role/hashicorp-manager/staging-github-action-runners-runner-role-toad and passing it via command: vault write \ auth/aws/role/master-role \ auth_type=iam \ policies=master-policy \ resolve_aws_unique_ids=false \ max_ttl=1h \ token_ttl=1h \ bound_iam_principal_arn=${runner_instance_profile_role}

Should allow me to login just like this:

h-4.2$ vault login -method=aws role=master-role Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token.

Key Value

token obfuscated token_accessor obfuscated token_duration 1h token_renewable true token_policies ["default" "master-policy"] identity_policies [] policies ["default" "master-policy"] token_meta_account_id obfuscated token_meta_auth_type iam token_meta_role_id obfuscated sh-4.2$

Environment:

Vault server configuration file(s):

ui = false

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file   = "obfuscated"
  tls_key_file    = "obfuscated"
}

service_registration "consul" {
  address = "127.0.0.1:8500"
}

          service_registration "consul" {
            address = "127.0.0.1:8500"
          }
          seal "awskms" {
            region     = "us-east-1"
            kms_key_id = "obfuscated"
          }

          storage "dynamodb" {
            ha_enabled = "true"
            region = "us-east-1"
            table  = "obfuscated"
          }
          # HA settings
          cluster_addr  = "https://obfuscated:8201"
          api_addr      = "https://obfuscated:8200"

telemetry {
  statsd_address = "localhost:8125"
  disable_hostname = true
}

Additional context Add any other context about the problem here.

heatherezell commented 2 years ago

Hello! Can you please try to replicate this issue with a newer version of Vault? You can see the Vault releases available here: https://releases.hashicorp.com/vault/ As a general rule, we accept issues for versions "n-2", so we would accept issues of bug reports for version 1.7 or later. Thanks!

T0tt1 commented 2 years ago

Hello! Can you please try to replicate this issue with a newer version of Vault? You can see the Vault releases available here: https://releases.hashicorp.com/vault/ As a general rule, we accept issues for versions "n-2", so we would accept issues of bug reports for version 1.7 or later. Thanks!

Thank you for the feedback, sir/madam!

Please find that the same issue has been observed after upgrading to version 1.9.0.

sh-4.2$ vault login -method=aws role=master-role
Error authenticating: context deadline exceeded
sh-4.2$ vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    obfuscated
Threshold                obfuscated
Version                  1.9.0
Storage Type             dynamodb
Cluster Name             vault-cluster
Cluster ID               obfuscated
HA Enabled               true
HA Cluster               https://obfuscated:8201
HA Mode                  active
Active Since             2021-12-16T11:03:58.195799046Z
heatherezell commented 2 years ago

Thanks! I'll have someone take a look.

heatherezell commented 2 years ago

Also, looking at some resources for this issue, there might be a problem with network connectivity. Can you run curl $VAULT_ADDR/v1/sys/seal-status from the node you're running the vault login from? You may also wish to increase the client timeout, like so: export VAULT_CLIENT_TIMEOUT=300s Let me know if either of those work for you!

T0tt1 commented 2 years ago

Also, looking at some resources for this issue, there might be a problem with network connectivity. Can you run curl $VAULT_ADDR/v1/sys/seal-status from the node you're running the vault login from? You may also wish to increase the client timeout, like so: export VAULT_CLIENT_TIMEOUT=300s Let me know if either of those work for you!

Thank you for your suggestion. There is no issue with the network connectivity. Please check below:

sh-4.2$ curl $VAULT_ADDR/v1/sys/seal-status
{"type":"shamir","initialized":true,"sealed":false,"t":obfuscated,"n":obfuscated,"progress":0,"nonce":"","version":"1.9.0","migration":false,"cluster_name":"vault-cluster","cluster_id":"obfuscated","recovery_seal":true,"storage_type":"dynamodb"}
sh-4.2$

Response is received immediately after executing the curl command hence no need to update the timeout period.

heatherezell commented 2 years ago

In some cases, we've seen where the request to AWS may be not responding within the timeout period, which is why I recommend trying to increase that value. Please give it a try and let me know if it works. :)

T0tt1 commented 2 years ago

In some cases, we've seen where the request to AWS may be not responding within the timeout period, which is why I recommend trying to increase that value. Please give it a try and let me know if it works. :)

Unfortunately proposed action is not working. I still think this is a bug and is not a misconfiguration on my end.

sh-4.2$ export PRIVATE_IP=$(dig +short obfuscated)
sh-4.2$ export VAULT_ADDR=https://$PRIVATE_IP:8200
sh-4.2$ vault login -method=aws header_value=$PRIVATE_IP role=master-role
Error authenticating: context deadline exceeded
sh-4.2$ export VAULT_CLIENT_TIMEOUT=300s
sh-4.2$ vault login -method=aws header_value=$PRIVATE_IP role=master-role
Error authenticating: Error making API request.

URL: PUT https://obfuscated:8200/v1/auth/aws/login
Code: 400. Errors:

* error looking up full ARN of entity &{aws obfuscated assumed-role  obfuscated i-03_obfuscated}: error fetching role "obfuscated": RequestError: send request failed
caused by: Post https://iam.amazonaws.com/: dial tcp obfuscated:443: i/o timeout
sh-4.2$
heatherezell commented 2 years ago

Can you confirm that you can reach iam.amazonaws.com:443 from your host?

T0tt1 commented 2 years ago

Can you confirm that you can reach iam.amazonaws.com:443 from your host?

I confirm that address is reachable.

sh-4.2$ telnet iam.amazonaws.com 443
Trying 52.46.155.37...
Connected to iam.amazonaws.com.
Escape character is '^]'.
heatherezell commented 2 years ago

Thanks, @T0tt1! I appreciate your patience. I'll see what we can find out from our side. :)

ocharles commented 2 weeks ago

Did anything come of this? I'm hitting exactly this issue, with all the same diagnosis

Edit: I found out my issue was that I had set

resource "vault_aws_auth_backend_role" "nomad_client" {
  ...
  inferred_aws_region  = "us-east-1c"
}

This should actually have been us-east-1