Description

I'm using AWS Glue as the output transformer of my Kinesis Firehose stream, I configured it to use always use the latest version from a Glue Table. The problem is that every time I release a new table version the kinesis stream definition also needs to be "touched", as if left without any changes it is not taking the updated table definition into account.

The solution I can see for this problem is to declare the Glue Table with an explicit version on Kinesis Firehose and this should do the trick, so every time I change the table definition a new version is generated and Firehose object is updated in the same plan.

Affected Resource(s) and/or Data Source(s)

aws_glue_catalog_table

Potential Terraform Configuration

The field aws_glue_catalog_table.storage_descriptor.version_id should be added as an auto generated field pointing to the latest version of the table declaration.

It could be used as the value for the definition of aws_kinesis_firehose_delivery_stream.extended_s3_configuration.data_format_conversion_configuration.schema_configuration.version_id.

resource "aws_glue_catalog_table" "glue_table" {
  name = "test_table"
  storage_descriptor {
    ....
  }
}
resource "aws_kinesis_firehose_delivery_stream" "firehose" {
  destination = "extended_s3"

  extended_s3_configuration {
    data_format_conversion_configuration {
      enabled = true
      "schema_configuration" {
        database_name = "${aws_glue_catalog_table.glue_table.dabase_name}"
        table_name = "${aws_glue_catalog_table.glue_table.name}"
        version_id = "${aws_glue_catalog_table.glue_table.storage_descriptor.version_id}"
      }
    }
  }
}

References

5906

Would you like to implement a fix?

None

hashicorp / terraform-provider-aws

[Enhancement]: Make latest AWS Glue table version available from attributes of aws_glue_catalog_table #39593