[bug] terraform plan fails with error reading wrappers from "recovery_kms_hcl" on v1.1.5

[justenwalker]

Terraform Version

v1.3.7

Provider Version

v1.1.5

Terraform Configuration Files

terraform {
  required_providers {
    boundary = {
      source  = "hashicorp/boundary"
      version = "1.1.5"
    }
  }
}
provider "boundary" {
  addr             = var.boundary_provider_addr
  recovery_kms_hcl = <<-EOT
kms "awskms" {
  purpose    = "recovery"
  region     = "us-west-2"
  kms_key_id = "92aa3cc0-37ed-4088-9d7d-373d512cbf05"
}
EOT
}

Expected Behavior

Terraform Plan should succeed

Actual Behavior

Terraform exits with:

│ Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: Unrecognized remote plugin message: 
│ This usually means
│   the plugin was not compiled for this architecture,
│   the plugin is missing dynamic-link libraries necessary to run,
│   the plugin is not executable by this process due to file permissions, or
│   the plugin failed to negotiate the initial go-plugin protocol handshake
│ 
│ Additional notes about plugin:
│   Path: /tmp/2588365193/boundary-plugin-kms-awskms-b2f6Z
│   Mode: -rwx------
│   Owner: 100 [atlantis] (current: 100 [atlantis])
│   Group: 1000 [atlantis] (current: 1000 [atlantis])
│   ELF architecture: EM_X86_64 (current architecture: amd64)
│ 
│ 
│   with provider["registry.terraform.io/hashicorp/boundary"],
│   on provider.tf line 29, in provider "boundary":
│   29: provider "boundary" {

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. configure boundary plugin version v1.1.5
2. terraform plan

Important Factoids

• Reproduced on via Atlantis running in ECS Fargate
• After reverting back to v1.1.4, I do not see this issue; so it seems specific to this build.

References

Seems similar to

• GH-282

[madianas21]

Hi! We are experiencing the same issue. Downgrading to version 1.1.4 fixes it.

│ Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: Unrecognized remote plugin message: 
│ This usually means
│   the plugin was not compiled for this architecture,
│   the plugin is missing dynamic-link libraries necessary to run,
│   the plugin is not executable by this process due to file permissions, or
│   the plugin failed to negotiate the initial go-plugin protocol handshake
│ 
│ Additional notes about plugin:
│   Path: /tmp/2105666918/boundary-plugin-kms-awskms-YDXWv
│   Mode: -rwx------
│   Owner: 100 [atlantis] (current: 100 [atlantis])
│   Group: 1000 [atlantis] (current: 1000 [atlantis])
│   ELF architecture: EM_X86_64 (current architecture: amd64)
│ 
│ 
│   with provider["registry.terraform.io/hashicorp/boundary"],
│   on providers.tf line 68, in provider "boundary":
│   68: provider "boundary" {

Our Providers file:

terraform {
  required_version = ">= 1.1.9"

  required_providers {
    ....
    boundary = {
      source  = "hashicorp/boundary"
      version = ">= 1.1.4"
    }
  }

provider "boundary" {
  addr             = "https://boundary.domain.com"
  recovery_kms_hcl = <<EOT
kms "awskms" {
    purpose    = "recovery"
  key_id     = "global_recovery"
  kms_key_id = "arn:aws:kms:eu-central-1:XXXXXXXXX:key/XXXXXXX-XXX-XXXX-XXXX-XXXXXXXXX"
  region     = "eu-central-1"
}
EOT
}

[micchickenburger]

I'm getting this with the new boundary provider version 1.1.8 as well.

[mikemountain]

@micchickenburger are you using Atlantis as well?

[micchickenburger]

@micchickenburger are you using Atlantis as well?

What is Atlantis?

[aruandre]

@mikemountain we're having the same issue with all versions starting from 1.1.3, didn't test with versions below that:

Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/568767209/boundary-plugin-kms-awskms-VfeXG: no such file or directory
│ 
│   with provider["registry.terraform.io/hashicorp/boundary"],
│   on providers.tf line 34, in provider "boundary":
│   34: provider "boundary" {

provider config:

provider "boundary" {
  addr             = local.url
  recovery_kms_hcl = <<EOT
    kms "awskms" {
        purpose    = "recovery"
        region = "${local.region}"
        kms_key_id = "${data.aws_kms_key.by_alias.id}"
        access_key = "${data.aws_ssm_parameter.iam_access_key.value}"
        secret_key = "${data.aws_ssm_parameter.iam_secret_access_key.value}"
    }
    EOT
}

[justenwalker]

I think I understand why this broke. Something with the new build in 1.1.5 introduced a change that is incompatible with musl; Atlantis default docker image is base on Alpine, which uses musl.

When I used the debian variant of Atlantis, it was able to upgrade to 1.1.9 and run my plan/apply

See also: https://discuss.hashicorp.com/t/error-reading-wrappers-from-recovery-kms-hcl/56591/3

[metanovii]

I think I understand why this broke. Something with the new build in 1.1.5 introduced a change that is incompatible with musl; Atlantis default docker image is base on Alpine, which uses musl.

When I used the debian variant of Atlantis, it was able to upgrade to 1.1.9 and run my plan/apply

See also: https://discuss.hashicorp.com/t/error-reading-wrappers-from-recovery-kms-hcl/56591/3

I confirm that the problem only occurs when using alpine

[MatthieuuC]

See https://github.com/hashicorp/terraform-provider-boundary/issues/255#issuecomment-1270997187

You should add libc6-compat into your docker image