awscc_timestream_scheduled_query is recreated during every apply

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
The resources and data sources in this provider are generated from the CloudFormation schema, so they can only support the actions that the underlying schema supports. For this reason submitted bugs should be limited to defects in the generation and runtime code of the provider. Customizing behavior of the resource, or noting a gap in behavior are not valid bugs and should be submitted as enhancements to AWS via the CloudFormation Open Coverage Roadmap.

Terraform CLI and Terraform AWS Cloud Control Provider Version

Terraform v1.5.0
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.8.0
+ provider registry.terraform.io/hashicorp/awscc v0.55.0

Affected Resource(s)

awscc_timestream_scheduled_query

Terraform Configuration Files

Code for reproducting is available here: https://gist.github.com/Mallear/43107fd180fd7f4b1266f503f02d2fdf#file-main-tf

Debug Output

https://gist.github.com/Mallear/43107fd180fd7f4b1266f503f02d2fdf#file-awscc_timestream_scheduled_query-2nd-apply

Expected Behavior

Terraform does not recreate the resource unless breaking changes were made.

Actual Behavior

Timestream scheduled query is well created at first but when re-applying the stack, Terraform find a lot of breaking changes and recreate the resource with the exact configuration.

By deleting the scheduled query from Terraform state and re-importing it, I see that all fields that "# forces replacement" are null. See https://gist.github.com/Mallear/43107fd180fd7f4b1266f503f02d2fdf#file-state_after_import-json

Steps to Reproduce

With the given code:

terraform apply
terraform apply

the debug log:

2024-02-15T20:45:55.216-0800 [WARN]  Provider "registry.terraform.io/hashicorp/awscc" produced an unexpected new value for awscc_timestream_scheduled_query.this during refresh.
      - .schedule_configuration: was cty.ObjectVal(map[string]cty.Value{"schedule_expression":cty.StringVal("rate(1 hour)")}), but now null
      - .scheduled_query_execution_role_arn: was cty.StringVal("arn:aws:iam::204034886740:role/TimestreamScheduledQueriesDebugRole"), but now null
      - .scheduled_query_name: was cty.StringVal("time-stream-scheduled-queries-debug-query"), but now null
      - .kms_key_id: was cty.StringVal("arn:aws:kms:us-east-1:204034886740:key/1cfbd15d-7e51-43b8-95c4-c05a607a86a2"), but now null
      - .notification_configuration: was cty.ObjectVal(map[string]cty.Value{"sns_configuration":cty.ObjectVal(map[string]cty.Value{"topic_arn":cty.StringVal("arn:aws:sns:us-east-1:204034886740:timestream-scheduled-queries-debug-errors.fifo")})}), but now null
      - .error_report_configuration: was cty.ObjectVal(map[string]cty.Value{"s3_configuration":cty.ObjectVal(map[string]cty.Value{"bucket_name":cty.StringVal("timestream-scheduled-queries-debug-h25csut6flcv"), "encryption_option":cty.NullVal(cty.String), "object_key_prefix":cty.NullVal(cty.String)})}), but now null
      - .target_configuration: was cty.ObjectVal(map[string]cty.Value{"timestream_configuration":cty.ObjectVal(map[string]cty.Value{"database_name":cty.StringVal("timestream-scheduled-queries-debug-db"), "dimension_mappings":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"dimension_value_type":cty.StringVal("VARCHAR"), "name":cty.StringVal("organization")}), cty.ObjectVal(map[string]cty.Value{"dimension_value_type":cty.StringVal("VARCHAR"), "name":cty.StringVal("workspace")})}), "measure_name_column":cty.StringVal("measure_name"), "mixed_measure_mappings":cty.NullVal(cty.List(cty.Object(map[string]cty.Type{"measure_name":cty.String, "measure_value_type":cty.String, "multi_measure_attribute_mappings":cty.List(cty.Object(map[string]cty.Type{"measure_value_type":cty.String, "source_column":cty.String, "target_multi_measure_attribute_name":cty.String})), "source_column":cty.String, "target_measure_name":cty.String}))), "multi_measure_mappings":cty.ObjectVal(map[string]cty.Value{"multi_measure_attribute_mappings":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"measure_value_type":cty.StringVal("BIGINT"), "source_column":cty.StringVal("total_clicks"), "target_multi_measure_attribute_name":cty.NullVal(cty.String)})}), "target_multi_measure_name":cty.NullVal(cty.String)}), "table_name":cty.StringVal("target"), "time_column":cty.StringVal("time_day")})}), but now null
      - .query_string: was cty.StringVal("SELECT organization, workspace, date_trunc('day', time) as time_day, 'clicks' as measure_name, sum(measure_value::bigint)as total_clicks FROM \"timestream-scheduled-queries-debug-db\".\"source\" group by organization, workspace, date_trunc('day', time) order by organization"), but now null

interesting point, on my test, I noticed two similar attributes with different name:

            "scheduled_query_execution_role_arn": 
            ...
            "sq_scheduled_query_execution_role_arn": "arn:aws:iam::204034886740:role/TimestreamScheduledQueriesDebugRole",

there are actually more than one attribute duplicates, this is just an example.

I am confident that AWSCC schema for timestream matched CCAPI :

aws cloudformation describe-type --type RESOURCE --type-name AWS::Timestream::ScheduledQuery | jq -r ".Schema" | jq ".properties" 
{
  "Arn": {
    "$ref": "#/definitions/Arn"
  },
  "ScheduledQueryName": {
    "$ref": "#/definitions/ScheduledQueryName"
  },
  "QueryString": {
    "$ref": "#/definitions/QueryString"
  },
  "ScheduleConfiguration": {
    "$ref": "#/definitions/ScheduleConfiguration"
  },
  "NotificationConfiguration": {
    "$ref": "#/definitions/NotificationConfiguration"
  },
  "ClientToken": {
    "$ref": "#/definitions/ClientToken"
  },
  "ScheduledQueryExecutionRoleArn": {
    "$ref": "#/definitions/ScheduledQueryExecutionRoleArn"
  },
  "TargetConfiguration": {
    "$ref": "#/definitions/TargetConfiguration"
  },
  "ErrorReportConfiguration": {
    "$ref": "#/definitions/ErrorReportConfiguration"
  },
  "KmsKeyId": {
    "$ref": "#/definitions/KmsKeyId"
  },
  "SQName": {
    "description": "The name of the scheduled query. Scheduled query names must be unique within each Region.",
    "type": "string"
  },
  "SQQueryString": {
    "description": "The query string to run. Parameter names can be specified in the query string @ character followed by an identifier. The named Parameter @scheduled_runtime is reserved and can be used in the query to get the time at which the query is scheduled to run. The timestamp calculated according to the ScheduleConfiguration parameter, will be the value of @scheduled_runtime paramater for each query run. For example, consider an instance of a scheduled query executing on 2021-12-01 00:00:00. For this instance, the @scheduled_runtime parameter is initialized to the timestamp 2021-12-01 00:00:00 when invoking the query.",
    "type": "string"
  },
  "SQScheduleConfiguration": {
    "description": "Configuration for when the scheduled query is executed.",
    "type": "string"
  },
  "SQNotificationConfiguration": {
    "description": "Notification configuration for the scheduled query. A notification is sent by Timestream when a query run finishes, when the state is updated or when you delete it.",
    "type": "string"
  },
  "SQScheduledQueryExecutionRoleArn": {
    "description": "The ARN for the IAM role that Timestream will assume when running the scheduled query.",
    "type": "string"
  },
  "SQTargetConfiguration": {
    "description": "Configuration of target store where scheduled query results are written to.",
    "type": "string"
  },
  "SQErrorReportConfiguration": {
    "description": "Configuration for error reporting. Error reports will be generated when a problem is encountered when writing the query results.",
    "type": "string"
  },
  "SQKmsKeyId": {
    "description": "The Amazon KMS key used to encrypt the scheduled query resource, at-rest. If the Amazon KMS key is not specified, the scheduled query resource will be encrypted with a Timestream owned Amazon KMS key. To specify a KMS key, use the key ID, key ARN, alias name, or alias ARN. When using an alias name, prefix the name with alias/. If ErrorReportConfiguration uses SSE_KMS as encryption type, the same KmsKeyId is used to encrypt the error report at rest.",
    "type": "string"
  },
  "Tags": {
    "$ref": "#/definitions/Tags"
  }
}

CCAPI GetResource however did not return some of the attributes, for example the ScheduledQueryExecutionRoleArn

aws cloudcontrol get-resource --type-name AWS::Timestream::ScheduledQuery --identifier "arn:aws:timestream:us-east-1:204034886740:scheduled-query/time-stream-scheduled-queries-debug-query-41fcfbf13a0a1f62" | jq -r ".ResourceDescription.Properties" | jq "."
aws cloudcontrol get-resource --type-name AWS::Timestream::ScheduledQuery --identifier "arn:aws:timestream:us-east-1:204034886740:scheduled-query/time-stream-scheduled-queries-debug-query-41fcfbf13a0a1f62" | jq -r ".ResourceDescription.Properties" | jq "."
{
  "SQScheduleConfiguration": "{ScheduleExpression: rate(1 hour)}",
  "SQNotificationConfiguration": "{SnsConfiguration: {TopicArn: arn:aws:sns:us-east-1:204034886740:timestream-scheduled-queries-debug-errors.fifo}}",
  "SQErrorReportConfiguration": "{S3Configuration: {BucketName: timestream-scheduled-queries-debug-h25csut6flcv,EncryptionOption: SSE_S3}}",
  "SQKmsKeyId": "arn:aws:kms:us-east-1:204034886740:key/1cfbd15d-7e51-43b8-95c4-c05a607a86a2",
  "SQQueryString": "SELECT organization, workspace, date_trunc('day', time) as time_day, 'clicks' as measure_name, sum(measure_value::bigint)as total_clicks FROM \"timestream-scheduled-queries-debug-db\".\"source\" group by organization, workspace, date_trunc('day', time) order by organization",
  "SQTargetConfiguration": "{TimestreamConfiguration: {DatabaseName: timestream-scheduled-queries-debug-db,TableName: target,TimeColumn: time_day,DimensionMappings: [{Name: organization,DimensionValueType: VARCHAR}, {Name: workspace,DimensionValueType: VARCHAR}],MultiMeasureMappings: {MultiMeasureAttributeMappings: [{SourceColumn: total_clicks,MeasureValueType: BIGINT}]},MeasureNameColumn: measure_name}}",
  "SQName": "time-stream-scheduled-queries-debug-query",
  "Arn": "arn:aws:timestream:us-east-1:204034886740:scheduled-query/time-stream-scheduled-queries-debug-query-41fcfbf13a0a1f62",
  "SQScheduledQueryExecutionRoleArn": "arn:aws:iam::204034886740:role/TimestreamScheduledQueriesDebugRole"
}

ScheduledQueryExecutionRoleArn apparently are createOnlyProperties , which mean the attribute value is not available during GetResource.

While this part is understood, but it didn't make sense that CCAPI introduces this as an attribute from the start.

I believe this is an upstream issue

hashicorp / terraform-provider-awscc