[Bug]: Unmarshal Response s3/GetObjectTagging failed, attempt 11/25, error SerializationError: failed to decode REST XML response

Terraform Core Version

1.3.7

AWS Provider Version

aws_v4.20.1_x5

Affected Resource(s)

aws_s3_bucket aws_s3_object

Expected Behavior

Our "wpad.dat" file should upload as our "helloworld.txt" file does.

Actual Behavior

We have an odd issue which is blocking us from moving forward, when we attempt to upload a "wpad.dat" (proxy config file) to a cloudflare r2 bucket (https://www.cloudflare.com/en-gb/products/r2/) which uses an "S3-compatible API".

In the terraform debug and in the cloudflare webUI the http post of the "wpad.dat" file to the cloudflare r2 bucket succeeds, but it looks like the http response to the terraform provider contains all or part of the content of the "wpad.dat" file containing a character which breaks the response. It attempts 25 times before we either have to cancel the pipeline or it times out.

If we use a another test file with only the text "helloworld" in it this works perfectly every time.

Relevant Error/Panic Output Snippet

N/A

Terraform Configuration Files

resource "aws_s3_bucket" "cf" {
  bucket = var.cloudflare_r2_bucket_name
}

resource "aws_s3_object" "cf" {
  bucket = var.cloudflare_r2_bucket_name
  key = var.cloudflare_r2_filename
  source = var.cloudflare_r2_filename
  etag = filemd5(var.cloudflare_r2_filename)
}

Steps to Reproduce

terraform apply using the wpad.dat will cause the problem.

Debug Output

This is a portion of the debug showing the break, I cannot share the "wpad.dat" publicly due to sensitive contents but the error does relate to portions of its contents eg: & (no semicolon), etc.,

2023-01-30T21:12:09.4704619Z 2023-01-30T21:12:09.467Z [DEBUG] provider.terraform-provider-aws_v4.20.1_x5: [aws-sdk-go] DEBUG: Unmarshal Response s3/GetObjectTagging failed, attempt 11/25, error SerializationError: failed to decode REST XML response
2023-01-30T21:12:09.4705833Z    status code: 200, request id: 
2023-01-30T21:12:09.4706541Z caused by: XML syntax error on line 21: invalid character entity & (no semicolon): timestamp=2023-01-30T21:12:09.467Z
2023-01-30T21:13:18.2283423Z ##[error]The operation was canceled.
2023-01-30T21:13:18.2306387Z ##[section]Finishing: Terraform configuration

Panic Output

N/A

Important Factoids

We think the solution is to not include the contents of the file in the http response.

References

We had previously logged this here https://github.com/aws/aws-sdk-go/issues/4709 and there are some similar feedbacks about the possible solution in the comments

hashicorp / terraform-provider-aws