Remote `terraform plan` fails when running on MacOS

[ticktockhouse]

Terraform Version

Terraform v1.9.2
on darwin_arm64
+ provider registry.terraform.io/digitalocean/digitalocean v2.38.0
+ provider registry.terraform.io/hashicorp/http v3.4.3
+ provider registry.terraform.io/hashicorp/tfe v0.55.0

Terraform Configuration Files

terraform {
  cloud {
    organization = "My Org"
    workspaces {
      name = "my-workspace"
    }
  }
}

I don't believe any other code is needed as it seems to happen with whatever config I'm using and whichever workspace I am using

Debug Output

https://gist.github.com/ticktockhouse/73397ae55fac0825490a6c5c82cd2331

Expected Behavior

terraform plan should trigger a remote plan

Actual Behavior

No plan happens and there does not seem to be any communication with HCP TF Cloud.

Steps to Reproduce

terraform plan

Additional Context

The exact same code works when running Terraform on Linux

References

No response

[crw]

Thanks for this report! If you are an HCP Terraform customer, please consider also filing a request through support via email tf-cloud@hashicorp.support or open a new request. Thanks!

[apparentlymart]

Thanks for reporting this!

Digging through all of the indirection in the code, is seems like what ultimately triggers this message about the connection to HCP Terraform is when a request fails in the way that go-tfe (the Go client library for the HCP Terraform API) considers retryable, which is decided by retryHTTPCheck.

The HCP Terraform integration in Terraform CLI does not appear to set retryServerErrors, which suggests that either attempts to make a connection to the HCP Terraform servers are not succeeding at all (preventing any HTTP request/response from occuring) or that the HTTP response code was 429 Too Many Requests.

The HCP Terraform integration doesn't seem to generate much trace log output itself so it's tough to say for sure what's happening here, but I don't see anything else in the trace log that's accessing https://app.terraform.io and so perhaps we could conclude that Terraform CLI is just, for some reason, completely unable to reach that port from your computer.

I'm not sure why that would be, and it might be hard to figure it out since we can't see how your computer or network are configured, but we've added the "cloud" label on this which will hopefully make it visible to the team that maintains the HCP Terraform integration who might have better ideas about how to debug this problem.

[apparentlymart]

Right after I posted the above I noticed something that contradicts my conclusion:

HTTP client GET request to https://app.terraform.io/.well-known/terraform.json

Terraform CLI apparently did manage to fetch the service discovery document for app.terraform.io, so it doesn't seem to be something as simple as failed connectivity between your client and that server. Whatever's going on here seems to be something specific to the HCP Terraform API client, encountered only after Terraform CLI already found the HCP Terraform API endpoints using service discovery.

[ticktockhouse]

Awesome engagement, people :) If you'd like me to run anything else this end to debug, I'd be more than happy to..