Request for Performance Insights in AWS Aurora RDS

[kami619]

Description

Request to enhance the Aurora DB cluster creation scripts with the option to enable performance insights based on a conditional flag such as AURORA_PERFORMANCE_INSIGHTS with the value being a boolean true/false.

We can achieve this in AWS CLI using a similar command and argument combo

aws rds create-db-cluster \
    --db-cluster-identifier my-aurora-cluster \
    --engine aurora-mysql \
    --master-username admin \
    --master-user-password secretPassword123 \
    --enable-performance-insights \
    --performance-insights-kms-key-id <your-kms-key-id> \
    --performance-insights-retention-period 7 \
    --region <your-region>

And below is an example for the Terraform resource module

resource "aws_rds_cluster_instance" "good_example" {
  name                 = "bar"
  performance_insights_enabled = true
  performance_insights_kms_key_id = "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
}

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/rds_cluster_instance#performance_insights_enabled

Optional: While we do this, we could explore if we do this per RDS node, if possible to contain the costs etc., as we might be interested in only the writer nodes or reader nodes based on the load test we perform.

Discussion

No response

Motivation

Enabling performance insights in AWS Aurora RDS nodes, will be beneficial to have on-demand additional performance observability on AWS Aurora instances. It will make the debugging faster to peek at slow queries and specific bottlenecks in the RDS instances.

Below are a couple of screenshots of the Performance Insights metrics from a load test run:

Details

Performance Insights is an advanced database performance monitoring feature that makes it easy to diagnose and solve performance challenges on Amazon RDS databases. It offers a free tier with 7 days of rolling data retention, and a paid long-term data retention option. For more information about Performance Insights, please visit the documentation

Performance Insights offers a rolling seven days of performance data history at no charge. This is enough time to troubleshoot a wide variety of issues. The Performance Insights free tier includes:

• 7 days of performance data history
• 1 million API requests per month

As an example, say you create a custom dashboard that makes calls to RDS Performance Insights’ two APIs, DescribeDimensionKeys, and GetResourceMetrics. The dashboard calls these two APIs every 5 seconds. That would be:

• 2 API calls (30 days 24 hours 60 minutes 60 seconds) / (5 seconds) = 1,036,800 API calls. The first 1M API calls are free per account and are $0.01 for each 1,000 calls after that. Thus, in this example the cost would be: ((1,036,800 - 1,000,000) / 1,000) * 0.01 = $0.37 per month
• If we monitored two instances with our custom dashboard refreshing every 5 seconds, the cost would be: ((2 1,036,800 - 1,000,000) / 1,000) 0.01 = $10.74 per month

[ahus1]

Thank you for suggesting this, I think it is great idea. If we are using dashboard in AWS, they would AFAIK only pull data when we're displaying them, so we should be fine.

If we choose to have a dashboard in Grafana, we would IMHO need to ensure that we use a datasource which pulls the metrics directly from AWS when we need them. If we would first import all data into Prometheus, this would incur the costs even if we don't look at the graph.

I would assume we would start with the default dashboard hosted on AWS?

[kami619]

correct @ahus1 the starting point would be to start off with the AWS dashboard, similar to how we are monitoring the Aurora RDS Cluster using Cloud Watch metrics. This would be an augmentation to our current approach. But its a good point to call out, the cost per metric would be different when we use a metric scraper like Prometheus.