[Bug]: aws_rds_cluster_activity_stream fails when starting database activity stream on Single-AZ database instances

[rmarilyak]

Terraform Core Version

1.7.4

AWS Provider Version

5.41.0

Affected Resource(s)

aws_rds_cluster_activity_stream

Expected Behavior

aws_rds_cluster_activity_stream resource successfully starts database activity streams for both RDS clusters and single DB Instances types

Actual Behavior

aws_rds_cluster_activity_stream resource is able to start database activity stream for single DB Instances (not clusters) - this can be seen in the AWS console, but fails with the error below

Relevant Error/Panic Output Snippet

╷
│ Error: waiting for RDS Cluster Activity Stream (arn:aws:rds:eu-central-1:XXXXXXXXXXXX:db:test-db) start: InvalidParameterValue: The provided ARN (arn:aws:rds:eu-central-1: XXXXXXXXXXXX:db:test-db) is invalid for this parameter (DBClusterIdentifier). Expected type = cluster, actual type = db
│   status code: 400, request id: 8324990f-97f6-4539-b0d2-7ad41673220f
│ 
│   with aws_rds_cluster_activity_stream.db_mssql_das,
│   on rds.tf line 120, in resource "aws_rds_cluster_activity_stream" "db_mssql_das":
│  120: resource "aws_rds_cluster_activity_stream" "db_mssql_das" {
│ 
╵

Terraform Configuration Files

module "db" {
  source  = "terraform-aws-modules/rds/aws"
  version = "6.5.2"

  identifier = "test-db"
  allocated_storage = 10
  max_allocated_storage = 20
  engine = "sqlserver-se"
  engine_version = "15.00.4073.23.v1"
  instance_class = "db.m5.large"
  multi_az = false
  storage_encrypted = true
  storage_type = "gp3"
  iops = 3000
  manage_master_user_password = false
  username = XXXXXX
  password = XXXXXX
...................
}

resource "aws_rds_cluster_activity_stream" "db_mssql_das" {
  resource_arn = module.db.db_instance_arn
  mode = "async"
  kms_key_id = aws_kms_key.db_mssql_das.key_id

  depends_on = [module.db]
}

resource "aws_kms_key" "db_mssql_das" {
  key_usage = "ENCRYPT_DECRYPT"
  enable_key_rotation = true
}

resource "aws_kms_alias" "db_mssql_das" {
  name = "alias/test-db-mssql-das"
  target_key_id = aws_kms_key.db_mssql_das.key_id
}

Steps to Reproduce

1. Create Single-AZ DB instance using terraform resources
2. Start database activity stream using aws_rds_cluster_activity_stream terraform resource

Debug Output

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_rds_cluster_activity_stream.db_mssql_das will be created
  + resource "aws_rds_cluster_activity_stream" "db_mssql_das" {
      + engine_native_audit_fields_included = false
      + id                                  = (known after apply)
      + kinesis_stream_name                 = (known after apply)
      + kms_key_id                          = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
      + mode                                = "async"
      + resource_arn                        = "arn:aws:rds:eu-central-1:XXXXXXXXXXXX:db:test-db"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions in workspace "test"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_rds_cluster_activity_stream.db_mssql_das: Creating...
aws_rds_cluster_activity_stream.db_mssql_das: Still creating... [10s elapsed]
aws_rds_cluster_activity_stream.db_mssql_das: Still creating... [20s elapsed]
aws_rds_cluster_activity_stream.db_mssql_das: Still creating... [30s elapsed]
╷
│ Error: waiting for RDS Cluster Activity Stream (arn:aws:rds:eu-central-1:XXXXXXXXXXXX:db:test-db) start: InvalidParameterValue: The provided ARN (arn:aws:rds:eu-central-1:XXXXXXXXXXXX:db:test-db) is invalid for this parameter (DBClusterIdentifier). Expected type = cluster, actual type = db
│   status code: 400, request id: 8324990f-97f6-4539-b0d2-7ad41673220f
│ 
│   with aws_rds_cluster_activity_stream.db_mssql_das,
│   on rds.tf line 120, in resource "aws_rds_cluster_activity_stream" "db_mssql_das":
│  120: resource "aws_rds_cluster_activity_stream" "db_mssql_das" {
│ 

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[acwwat]

@rmarilyak This is probably a usage error, as database activity stream is an Aurora feature, not regular RDS. Per your configuration, you are trying to use this feature against an RDS for SQL Server instance which is not supported AFAIK. Please double check your use case.

[rmarilyak]

@acwwat, that is not correct, because database Activity Streams supports Amazon RDS for SQL Server and Oracle as well.

Moreover, as I've mentioned above, the aws_rds_cluster_activity_stream terraform resource successfully starts the database activity stream even for RDS for SQL Server instance, but it fails with the above mentioned error afterwards. I am pretty much sure that it simply uses aws rds start-activity-stream CLI command 'under its hood', which successfully works with any most of RDS instances and engines (SQL Server as well). The only problem as I see it - the aws_rds_cluster_activity_stream terraform resource is simply can't handle the situations, when RDS instance does not have cluster ARN property, which is exactly my case.

[acwwat]

@rmarilyak Thanks for the additional context. According to the aws-sdk-go-v2 RDS changelog, the support was added since 2023-02-15, so it would have worked if the API supports it. The StartActivityStream API reference also suggests the error we are seeing that the resource ARN is for (Aurora) DB clusters, which along with the error and resource name prompted my original comment. So the two confusing piece of information, AWS documentation and the TF resource name, should be fixed at some point.

Looking more into the resource code, I see that it tries to find an activity stream on an read operation but it is hardcoded to handle a DB cluster. The create operation does a read in the end, so the creation might be failing because of the read instead of the StartActivityStream request. The actual behavior your described seems to suggest the same. The fix is probably to detect whether the ARN is for a cluster or instance and fetch the appropriate resource to read the activity stream settings.

[rmarilyak]

So, is there gonna be any solution to this?

[acwwat]

AWS API reference issue has also been reported to AWS as a support case. They have acknowledged the issue and will fix it "in due time" with no ETA nor notification when it is done. Just need to keep and eye out for the fix.