provider DNS failure on macOS on VPN

This is similar, if not identical, to https://github.com/hashicorp/terraform-provider-kubernetes/issues/969. More discussion about the underlying GOLANG issue as it relates to Terraform core https://github.com/hashicorp/terraform/issues/3536

In summary, when connected to a VPN on a Mac, the consul provider is bypassing the local resolver configuration, and fails DNS lookups like consul.service.consul. This is a pending issue with Vault, and was previously an issue in Terraform core, but has since been fixed. We know Terraform is fixed because Terraform is able to handle a backend configured to use consul.service.consul, and the problem described here only happens when using the consul provider.

Terraform Version

Terraform v1.0.5 on macOS 11.5.2 consul provider 2.13.0

Affected Resource(s)

Affects the provider generally, but not terraform itself.

Terraform Configuration Files

terraform {

  required_version = ">= 1.0"

  backend "consul" {
    address = "consul.service.consul:8500"
    path    = "tfdocs/dns-testing/state"
  }

  required_providers {
    consul = {
      version = ">= 2.13.0"
    }
}

provider "consul" {
  address = "consul.service.consul:8500"
}

data "consul_keys" "client_config" {
  key {
    name = "config"
    path = "some/path"
  }
}

Expected Behavior

The provider should be able to resolve the name consul.service.consul

Actual Behavior

DNS lookup fails, but only for the provider.

│ Error: Failed to read Consul key 'some/path': Get "http://consul.service.consul:8500/v1/kv/some/path": dial tcp: lookup consul.service.consul on 192.168.3.7:53: no such host

Here, 192.168.3.7 is the local DNS server, not the DNS configured for and on the other side of the VPN tunnel. The OS configuration scutil --dns shows correct:

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : ktzr.local
  nameserver[0] : 192.168.3.7
  if_index : 4 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  search domain[0] : mycorp01.com
  search domain[1] : node.consul
  search domain[2] : service.consul
  search domain[3] : mycorp02.com
  nameserver[0] : 10.10.0.4
  nameserver[1] : 10.10.0.5
  nameserver[2] : 10.11.0.5
  if_index : 13 (utun10)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

terraform plan

Important Factoids

If I recompile the provider locally (no changes necessary), it seems to work fine. At least in this case, setting CGO_ENABLED to 0 or 1 in GNUMakefile, or leaving it out and compiling the code as is, doesn't change anything. Just recompiling seems to somehow fix it.

hashicorp / terraform-provider-consul