Latest version of vault provider does not work with STS credentials with latest version of vault

Shocktrooper commented 1 year ago

Versions

Terraform version: 1.4.2 Vault: 1.13.0 Vault provider version: 3.14.0

Affected Resource(s)

vault_aws_access_credentials datasource

Terraform Configuration Files

provider "vault" {
  address          = "<Vault URL>"
  skip_child_token = true
  token            = "<Token>"
}

data "vault_aws_access_credentials" "billingCreds" {
  backend  = "devops-aws-admin"
  role     = "devops-admin-ci-role"
  type     = "sts"
  ttl      = "1h" # behavior is still the same if this is omitted
  region   = "us-east-1"
  role_arn = "arn:aws:iam::123456789:role/devops-account-management"
}

terraform {
  required_version = ">= 1.4.2"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.59.0"
    }
    vault = {
      source  = "hashicorp/vault"
      version = "3.14.0"
    }
  }
}

output "creds" {
  value = {
    access_key = data.vault_aws_access_credentials.billingCreds.access_key
    secret_key = data.vault_aws_access_credentials.billingCreds.secret_key
    token      = data.vault_aws_access_credentials.billingCreds.security_token
  }
  sensitive = true
}

Debug Output

https://gist.github.com/Shocktrooper/e2ca23783e2f62dfa3bcbad75e55e771

State file after above sample code above reports success

https://gist.github.com/Shocktrooper/c92638a01603efedcdab8ab4c22ebe9c

Expected Behavior

AWS credentials should be returned for the role specified in the datasource NOTE: Going through the UI returns valid credentials this behavior is only observed via the vault terraform provider

Actual Behavior

The plan/apply specifies that everything has went smoothly until the credentials are ready to be used. Looking at the output/plan null values are recorded in the state. When looking at the debug output an EOF is observed

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

terraform apply
terraform output creds If using the sample code

Important Factoids

Vault can lease the credentials to a user just fine that logs in via the UI and inputs the required fields. I have only observed this happening with the provider. I have not tried using the Vault CLI or a curl request to see if I can replicate it that way. Note: The provider works with our previous version of vault 1.12.1

Shocktrooper commented 1 year ago

Later today I will try to see if I can replicate this with a curl/vault CLI request

Shocktrooper commented 1 year ago

Postman script was a success the following is a sanitized response

{
    "request_id": "123456789-7479-4a2b-22c5-123456789",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": {
        "access_key": "123456789",
        "arn": "arn:aws:sts::123456789:assumed-role/devops-shsvc-account-management/vault-root-devops-admin-ci-role-123456789-123456789",
        "secret_key": "123456789/123456789",
        "security_token": "123456789",
        "ttl": 3599
    },
    "wrap_info": null,
    "warnings": null,
    "auth": null
}

robcasey commented 1 year ago

I'm experiencing this exact issue with

Terraform: 1.4.4 Vault: 1.13.1 Vault provider: 3.14.0

I've also tried several other versions of the provider including 3.13.0, 3.11.0, and 3.0.0 with the same results. Vault CLI requests work fine and return non-null values.

alex-ikse commented 1 year ago

https://github.com/hashicorp/terraform-provider-vault/blob/09dc5399b590ef1e58cc00f0ca8548bbf1fcb0b5/vault/data_source_aws_access_credentials.go#L174

the lease_id is used as ID for the terraform object... so, no lease_id (since v1.13), no object :/

see: https://github.com/hashicorp/vault/pull/15869

Shocktrooper commented 1 year ago

What would be a working & unique value to replace the lease id with?

Related issues/PR's

What caused this issue in Vault: https://github.com/hashicorp/vault/pull/15869
Other issues for end users not provider related : https://github.com/hashicorp/vault/issues/19513
Addition of no lease in vault for sts credentials but will not solve the provider issue as we need a unique value for the datasource per call: https://github.com/hashicorp/vault/pull/19923
PR to revert the vault changes: https://github.com/hashicorp/vault/pull/20034

alex-ikse commented 1 year ago

What would be a working & unique value to replace the lease id with?

Why not just data.access_key ?

kadamwolfe commented 1 year ago

We are experiencing the same issue with 1.13.x. Dynamic creds from the AWS mount aren't usable with assumed roles across accounts.

vinay-gopalan commented 1 year ago

Thanks for opening this issue! The updates made to STS leases in Vault 1.13 are in the process of being reverted; this would mean upcoming releases of Vault will go back to returning lease information like before. We apologize for the inconveniences caused during this transition! 🙏🏼

I'll be sure to update this issue thread as soon as the upcoming Vault patch version is released. We appreciate your feedback and patience!

mlcooper commented 1 year ago

@vinay-gopalan this has been released, correct?

mxk commented 1 month ago

I just ran into the same issue. My company is running a modified version of Vault and I think a recent change caused it to start returning empty lease_ids for STS creds. I'm working with that team to fix it on their side, but it would be great if this could be handled by the provider as well. It should be a tiny change to use the access key either unconditionally or when the lease_id is empty. Happy to submit a PR if that would be deemed acceptable.

hashicorp / terraform-provider-vault