terraform doesn't respect macOS /etc/resolver despite consul prescribing it's use

[kevinclark]

The context here is that I've setup a consul cluster in our datacenter and added a dns resolver to my local mac at /etc/resolver:

❯ cat /etc/resolver/consul
nameserver 10.100.1.196
port 8600

While system services all use the resolver just fine, terraform doesn't. The problem appears similar to this docker issue. This is particularly problematic as I'm using the Hashicorp-prescribed way to setup local dns resolution on a mac.

Terraform Version

Terraform v0.13.5

Terraform Configuration Files

terraform {
  backend "consul" {
    address = "consul.service.consul:8500"
    scheme = "http"
    path = "infra/nomad/terraform_state"
  }
}

Debug Output

❯ TF_LOG=trace terraform init
2020/10/29 10:07:13 [INFO] Terraform version: 0.13.5
2020/10/29 10:07:13 [INFO] Go runtime version: go1.14.10
2020/10/29 10:07:13 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init"}
2020/10/29 10:07:13 [DEBUG] Attempting to open CLI config file: /Users/kevinclark/.terraformrc
2020/10/29 10:07:13 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/10/29 10:07:13 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/10/29 10:07:13 [DEBUG] ignoring non-existing provider search directory /Users/kevinclark/.terraform.d/plugins
2020/10/29 10:07:13 [DEBUG] ignoring non-existing provider search directory /Users/kevinclark/Library/Application Support/io.terraform/plugins
2020/10/29 10:07:13 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2020/10/29 10:07:13 [INFO] CLI command args: []string{"init"}

2020/10/29 10:07:13 [TRACE] Meta.Backend: built configuration for "consul" backend with hash value 415441452
2020/10/29 10:07:13 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2020/10/29 10:07:13 [DEBUG] New state was assigned lineage "e47746cb-6665-287f-5e08-f67c7e37f50f"
2020/10/29 10:07:13 [TRACE] Meta.Backend: moving from default local state only to "consul" backend
Initializing the backend...
2020/10/29 10:07:13 [DEBUG] checking for provisioner in "."
2020/10/29 10:07:13 [DEBUG] checking for provisioner in "/usr/local/bin"
2020/10/29 10:07:13 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2020/10/29 10:07:13 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2020/10/29 10:07:13 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2020/10/29 10:07:13 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2020/10/29 10:07:13 [TRACE] statemgr.Filesystem: read nil snapshot
2020/10/29 10:07:13 [TRACE] Meta.Backend: ignoring local "default" workspace because its state is empty
2020/10/29 10:07:13 [DEBUG] New state was assigned lineage "5afef581-f4c4-fd50-eaa2-5616070b9063"
2020/10/29 10:07:13 [TRACE] Preserving existing state lineage "5afef581-f4c4-fd50-eaa2-5616070b9063"

Successfully configured the backend "consul"! Terraform will automatically
use this backend unless the backend configuration changes.

Error: Failed to get existing workspaces: Get "http://consul.service.consul:8500/v1/kv/infra/nomad/terraform_state-env:?keys=&separator=%2F": dial tcp: lookup consul.service.consul on 1.1.1.1:53: no such host

Expected Behavior

My consul agent (defined in /etc/resolver/consul) should have been used to resolve consul.service.consul, rather than going through my standard DNS servers.

Actual Behavior

The resolver was ignored.

Steps to Reproduce

1. Setup a consul resolver per these docs.
2. Configure a consul backend that uses a .consul address for the terraform.backend.address (see my config).
3. terraform init

Additional Context

Context is largely up top. I'm also on a VPN to get to my cluster, but that doesn't appear to effect the system resolution. The VPN also explicitly doesn't capture all traffic or rewrite DNS, so I believe it to be a moot point.

References

Docker seemed to have a similiar issue here.

[kevinclark]

Just found this issue which suggests it's a known issue. Seems very strange that consul would say "the resolver is the way to go" and then the hashistack would explicitly not use that supported method.

[danieldreier]

Hi @kevinclark - thanks for pointing this out, and for highlighting the gap between the two tools. Our intent, of course, is for these tools to work well together. I think that this is effectively a specific case of https://github.com/hashicorp/terraform/issues/3536 which is a long-standing issue. I'm going to close this issue as a duplicate of that, unless you strongly object, because the technical fix on our end is exactly the same.

I really appreciate you pointing this out from this new perspective. I've reached out to our internal release engineering team to highlight the fact that there is a cross-product issue with a probably-similar solution (cgo-enabled builds) and see when they can fit it in to their roadmap.

Please feel free to reach out (ddreier@hashicorp.com) if you feel strongly this shouldn't be closed as a duplicate - I'm happy to talk.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.