[Bug]: Dynamic Provider in Agents Broken

[hazam]

Terraform Core Version

1.4.2

AWS Provider Version

5.65.0

Affected Resource(s)

(probably) Dynamic Provider Credentials when using hosted agents (1.15.4 in our case). Reverting to 5.63.1 solves the issue.

Expected Behavior

Provider is autenticated correctly even when executed in hosted agents

Actual Behavior

Provider is not autenticated when executed in hosted agents and fails.

Relevant Error/Panic Output Snippet

╷
│ Error: No valid credential sources found
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on providers.tf line 2, in provider "aws":
│    2: provider "aws" {
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ Error: failed to refresh cached credentials, failed to retrieve
│ credentials, operation error STS: AssumeRoleWithWebIdentity, exceeded
│ maximum number of attempts, 3, https response error StatusCode: 0,
│ RequestID: , request send failed, Post
│ "https://sts.eu-south-1.amazonaws.com/": net/http: TLS handshake timeout

Reverting to 5.63.1 solves the issue.

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version ~> "5.0"
    }
  }
  required_version = "1.4.2"
}

provider "aws" {
  region = "eu-south-1"
}

Steps to Reproduce

Start a plan with dynamic provider configured inside a hosted agent (1.15.4)

Debug Output

Terraform v1.4.2
on linux_amd64
Initializing plugins and modules...
{"@level":"info","@message":"Terraform 1.4.2","@module":"terraform.ui","@timestamp":"2024-09-02T09:43:50.588414Z","terraform":"1.4.2","type":"version","ui":"1.1"}
{"@level":"info","@message":"Plan: 0 to add, 0 to change, 0 to destroy.","@module":"terraform.ui","@timestamp":"2024-09-02T09:44:44.231660Z","changes":{"add":0,"change":0,"remove":0,"operation":"plan"},"type":"change_summary"}
{"@level":"error","@message":"Error: No valid credential sources found","@module":"terraform.ui","@timestamp":"2024-09-02T09:44:44.231849Z","diagnostic":{"severity":"error","summary":"No valid credential sources found","detail":"Please see https://registry.terraform.io/providers/hashicorp/aws\nfor more information about providing credentials.\n\nError: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post \"https://sts.eu-south-1.amazonaws.com/\": net/http: TLS handshake timeout\n","address":"provider[\"registry.terraform.io/hashicorp/aws\"]","range":{"filename":"providers.tf","start":{"line":2,"column":16,"byte":83},"end":{"line":2,"column":17,"byte":84}},"snippet":{"context":"provider \"aws\"","code":"provider \"aws\" {","start_line":2,"highlight_start_offset":15,"highlight_end_offset":16,"values":[]}},"type":"diagnostic"}
Operation failed: failed running terraform plan (exit 1)

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @hazam 👋 Thank you for taking the time to raise this! This is a bit of an interesting scenario, because the provider doesn't do anything special in order to work with Dynamic Provider credentials. That feature is something built into HCP Terraform that ultimately just cleverly utilizes how the provider works in general.

With that in mind, I would recommend that you file a support ticket with the HCP Terraform team so that they can try to track down the root cause of this. Looking over the provider's changelog, the only thing that sticks out to me as a potential cause is that the provider was upgraded to use Go 1.23. We don't anticipate that the Go upgrade would cause any downstream issues, but stranger things have happened.

I'll keep this issue open for now as well, in case my initial thought are incorrect.

[hazam]

Done, support request #160016 on HCP side. Will keep you posted

[ewbankkit]

@hazam Assuming that the Go 1.22.6 downgrade with Terraform AWS Provider v5.67.0 fixed this problem, I'm going to close this issue. Discussion will continue in https://github.com/hashicorp/terraform-provider-aws/issues/39311.

[github-actions[bot]]

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.