CloudFormation stack update triggers

acdha commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

I have an upstream CloudFormation template source which periodically updates without changing the template URL. It would be nice if there was a way to trigger an update based on something like an ETag from the HTTP data provider, whether that's using the CloudFormation API's client request token concept or simply having a way to specify a value which is stored in the state file and triggers an update any time it changes.

New or Affected Resource(s)

aws_cloudformation_stack

Potential Terraform Configuration

resource "aws_cloudformation_stack" "MyStack" {
  name                 = "MyStack"
  tags                 = var.tags
  template_url         = var.mystack_template_url
  client_request_token = data.http.mytemplate.response_headers["ETag"]
}

References

https://support.hashicorp.com/hc/requests/29686

jgrumboe commented 3 years ago

Hi @acdha

I don't have the same problem, but reading through your description made me think about a workaround. Could the http datasource https://registry.terraform.io/providers/hashicorp/http/latest/docs/data-sources/http help you to solve your problem? You could use template_body instead of template_url and assign it the data.http.name.body. So each time the http datasource body changes your stack will do also. Just a thought, haven't tested it. Cheers, Johannes

acdha commented 3 years ago

Hi @acdha

I don't have the same problem, but reading through your description made me think about a workaround. Could the http datasource https://registry.terraform.io/providers/hashicorp/http/latest/docs/data-sources/http help you to solve your problem? You could use template_body instead of template_url and assign it the data.http.name.body. So each time the http datasource body changes your stack will do also. Just a thought, haven't tested it. Cheers, Johannes

This can work in some cases but there's an unfortunate inconsistency which I see I mentioned on the internal ticket but not here: using template_body limits your template to a maximum size of 51,200 bytes but template_url allows files as large as 460,800 bytes. There was a separate issue with the limited set of Content-Types supported by the http provider but that was resolved in https://github.com/hashicorp/terraform-provider-http/pull/16 in the 2.0.0 release.

jgrumboe commented 3 years ago

@acdha I see what you mean regarding the size limits of template_body and template_url (containing the body). Didn't check the API docs before.

prgres commented 2 years ago

This feature would help me too

cacack commented 2 years ago

From this issue, and others, it definitely appears like a missing feature of the aws_cloudformation_stack resource. We're in the unfortunate boat of having to deploy a CloudFormation template (vs just converting it to Terraform). The template is being uploaded to an S3 bucket using aws_s3_object, and does change frequently. Our aws_cloudformation_stack resource doesn't detect the template change and therefore never updates.

This is on Terraform v1.1.9 and AWS provider v4.10.0.

An example HCL is below. This include our attempt at using depends_on to force aws_cloudformation_stack to detect the change.

resource "aws_s3_bucket" "mybucket" {
  bucket = "mybucket"
}

resource "aws_s3_object" "mytemplate" {
  bucket   = aws_s3_bucket.mybucket.id
  key      = "mytemplate.template"
  source   = "mytemplate.template"

  etag = filemd5("mytemplate.template")
}

resource "aws_cloudformation_stack" "mystack" {
  name         = "mystack"
  template_url = "https://${aws_s3_bucket.mybucket.bucket_regional_domain_name}/${aws_s3_object.mytemplate.id}"
  depends_on = [
    aws_s3_object.mytemplate,
  ]
}

asaub commented 1 year ago

Has anyone found a good solution for this issue?

We also have to manually update Stacks that use a template_url.

jgrumboe commented 1 year ago

I use the workaround to add some custom/random query parameter to the S3 URL. I either change it every now and then or one could also use https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/rotating to generate a time-based string. This query param is ignored by S3, but makes TF and CF think it's a new template_url.

briceburg commented 1 year ago

I implemented a workaround for this that modifies the template_url to include the ETag/shasum of the URL contents. The http data source is used to discover ETag. E.g.

data "http" "template" {
  url    = "https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml"
  method = "HEAD"
}

resource "aws_cloudformation_stack" "datadog_forwarder" {
  name         = "datadog-forwarder"
  ...
  template_url = "${data.http.template.url}?z=${trim(data.http.template.response_headers.Etag, "\"")}"
}

the change plan looks something like;

# module.aws-account-baseline.module.region-us-east-1.aws_cloudformation_stack.datadog_forwarder will be updated in-place
  ~ resource "aws_cloudformation_stack" "datadog_forwarder" {
      ~ template_url       = "https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml?z=\"89baed7003af9300aa1a5707b66fb67c\"" -> "https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml?z=89baed7003af9300aa1a5707b66fb67c"
        # (7 unchanged attributes hidden)
    }

Mike-Nahmias commented 2 months ago

@briceburg Does this still work for you? For some reason when I try this the plan shows what you described, but Terraform escapes the special characters in the URL before trying to download the template, which causes S3 to give an AccessDenied error.

For example, the URL in my error would be:

https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml%3Fz%3Dasdf

I think this may be an issue with aws_cloudformation_stack_set but not aws_cloudformation_stack

hashicorp / terraform-provider-aws