search

hashicorp / terraform-provider-vault

Terraform Vault provider
https://www.terraform.io/docs/providers/vault/
Mozilla Public License 2.0
457 stars 536 forks source link

[Bug]: Provider crashes when you want to read multiple resources, but OIDC fails to provide a token before timing out #2131

Open danielskowronski opened 7 months ago

danielskowronski commented 7 months ago

Terraform Core Version

1.7.0

Terraform Vault Provider Version

3.24.0

Vault Server Version

1.13.3

Affected Resource(s)

At least:

Expected Behavior

When using OIDC to login and reading more than one vault_kv_secret_v2, the provider should gracefully time out on waiting to obtain a token.

Actual Behavior

It crashes in a nasty way.

Relevant Error/Panic Output Snippet

data.vault_kv_secret_v2.some_secret: Reading...
data.vault_kv_secret_v2.some_other_secret: Reading...
data.vault_kv_secret_v2.some_other_secret: Still reading... [10s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [10s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [20s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [20s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [30s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [30s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [40s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [40s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [50s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [50s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m0s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m0s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m10s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m10s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m20s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m20s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m30s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m30s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m40s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m40s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m50s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m50s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [2m0s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [2m0s elapsed]
╷
│ Error: Request cancelled
│
│   with data.vault_kv_secret_v2.some_secret,
│   on test.tf line 19, in data "vault_kv_secret_v2" "some_secret":
│   19: data "vault_kv_secret_v2" "some_secret" {
│
│ The plugin.(*GRPCProvider).ReadDataSource request was cancelled.
╵
╷
│ Error: Timed out waiting for response from provider
│
│   with data.vault_kv_secret_v2.some_other_secret,
│   on test.tf line 23, in data "vault_kv_secret_v2" "some_other_secret":
│   23: data "vault_kv_secret_v2" "some_other_secret" {
│
╵

Stack trace from the terraform-provider-vault_v3.24.0_x5 plugin:

panic: http: multiple registrations for /oidc/callback

goroutine 39 [running]:
net/http.(*ServeMux).Handle(0x104c82c40, {0x10393e25d, 0xe}, {0x1041b1220?, 0x14000a89540})
    net/http/server.go:2530 +0x1f8
net/http.(*ServeMux).HandleFunc(...)
    net/http/server.go:2567
net/http.HandleFunc(...)
    net/http/server.go:2579
github.com/hashicorp/vault-plugin-auth-jwt.(*CLIHandler).Auth(0x14000d0a750?, 0x1400074a500, 0x14000d0aea0)
    github.com/hashicorp/vault-plugin-auth-jwt@v0.18.0/cli.go:125 +0x430
github.com/hashicorp/terraform-provider-vault/internal/provider.(*AuthLoginOIDC).Login(0x14000d0a750?, 0x103936d6f?)
    github.com/hashicorp/terraform-provider-vault/internal/provider/auth_oidc.go:116 +0x6c
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).setClient(0x14000a507c0)
    github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:292 +0xb38
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).getClient(...)
    github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:403
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).GetClient(0x14000a507c0)
    github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:64 +0x70
github.com/hashicorp/terraform-provider-vault/internal/provider.GetClient({0x104177760?, 0x14000c30100?}, {0x104101c80?, 0x14000a507c0})
    github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:488 +0x2ac
github.com/hashicorp/terraform-provider-vault/vault.kvSecretV2DataSourceRead({0x1041c0f58?, 0x14000c04450?}, 0x104c83500?, {0x104101c80?, 0x14000a507c0?})
    github.com/hashicorp/terraform-provider-vault/vault/data_source_kv_secret_v2.go:94 +0x44
github.com/hashicorp/terraform-provider-vault/vault.kvSecretV2DataSource.ReadContextWrapper.func1({0x1041c1000, 0x14000b251f0}, 0x0?, {0x104101c80, 0x14000a507c0})
    github.com/hashicorp/terraform-provider-vault/internal/provider/provider.go:259 +0x70
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x1400019dce0, {0x1041c0f58, 0x14000c04450}, 0xd?, {0x104101c80, 0x14000a507c0})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.31.0/helper/schema/resource.go:795 +0xe8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ReadDataApply(0x1400019dce0, {0x1041c0f58, 0x14000c04450}, 0x14000c30000, {0x104101c80, 0x14000a507c0})
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.31.0/helper/schema/resource.go:1015 +0x124
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadDataSource(0x14000864b10, {0x1041c0f58?, 0x14000c04360?}, 0x14000916020)
    github.com/hashicorp/terraform-plugin-sdk/v2@v2.31.0/helper/schema/grpc_provider.go:1251 +0x2e0
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadDataSource(0x14000a8fd60, {0x1041c0f58?, 0x140008692c0?}, 0x14000066780)
    github.com/hashicorp/terraform-plugin-go@v0.20.0/tfprotov5/tf5server/server.go:699 +0x324
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadDataSource_Handler({0x104159b60?, 0x14000a8fd60}, {0x1041c0f58, 0x140008692c0}, 0x14000b2ed80, 0x0)
    github.com/hashicorp/terraform-plugin-go@v0.20.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:539 +0x164
google.golang.org/grpc.(*Server).processUnaryRPC(0x140002ae800, {0x1041c0f58, 0x14000869230}, {0x1041c7340, 0x14000182680}, 0x140009aec60, 0x14000868bd0, 0x104c6e488, 0x0)
    google.golang.org/grpc@v1.60.1/server.go:1372 +0xb8c
google.golang.org/grpc.(*Server).handleStream(0x140002ae800, {0x1041c7340, 0x14000182680}, 0x140009aec60)
    google.golang.org/grpc@v1.60.1/server.go:1783 +0xc44
google.golang.org/grpc.(*Server).serveStreams.func2.1()
    google.golang.org/grpc@v1.60.1/server.go:1016 +0x5c
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 26
    google.golang.org/grpc@v1.60.1/server.go:1027 +0x138

Error: The terraform-provider-vault_v3.24.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Terraform Configuration Files

terraform {
  required_providers {
    vault = {
      source  = "hashicorp/vault"
      version = "3.24.0"
    }
  }

  required_version = "1.7.0"
}

provider "vault" {
  address = "https://..."
  auth_login_oidc {
    role = "default"
  }
}

data "vault_kv_secret_v2" "some_secret" {
  mount = "..."
  name  = "..."
}
data "vault_kv_secret_v2" "some_other_secret" {
  mount = "..."
  name  = "..."
}

Steps to Reproduce

  1. Use provider with auth_login_oidc
  2. Declare at least two data sources of type vault_kv_secret_v2. If you use just one, it does work, so it's likely some race condition.
  3. Run terraform apply or terraform plan
  4. Browser gets opened to log you in and the session is expired (so SSO does not automatically redirect you back)
  5. Do nothing there
  6. After 2 minutes, the provider crashes in a nasty way

Debug Output

No response

Panic Output

No response

Important Factoids

Users are very likely to issue terraform apply and miss the fact they had a new browser tab opened - it can mix up with the time they opened a new tab or for larger deployments they can just go to grab a coffee.

References

No response

Would you like to implement a fix?

None

danielskowronski commented 7 months ago

The logic in https://github.com/hashicorp/terraform-provider-vault/blob/main/internal/provider/meta.go#L268 seems to have some bug. Likely, the cloned client is left in an unclean state after timeout from TF (registered handler for /oidc/callback) and there's some missing check to see if it can be re-used so an unsafe authLogin.Login(clone) is issued.

Or it can be caused by the library used, https://github.com/hashicorp/vault-plugin-auth-jwt/pull/115 seems to confirm this behaviour.

fairclothjm commented 7 months ago

@danielskowronski Hi, thanks for reporting! I believe this does need to be fixed in vault-plugin-auth-jwt. We will see if we can get this resolved.