Open sastels opened 1 month ago
Roughly, we're needing an additional GB every month. Here's how low the local storage has been since we started this dataset~(~'...~'DBInstanceIdentifier~'notification-canada-ca-production-instance-0~(label~'notification-canada-ca-production-instance-0~region~'ca-central-1~visible~false))~(~'...~'notification-canada-ca-production-instance-1~(label~'notification-canada-ca-production-instance-1~region~'ca-central-1~visible~false))~(~'...~'notification-canada-ca-production-instance-2~(label~'notification-canada-ca-production-instance-2~region~'ca-central-1~visible~false)))~period~86400~region~'ca-central-1~stat~'Minimum~title~'FreeLocalStorage~yAxis~(left~(min~0))~start~'2024-04-20T043a003a00.000Z~end~'2024-07-16T033a593a59.000Z~view~'timeSeries~stacked~false))
Describe the bug
The production
notifications
dataset in QuickSight is failing during its nightly creation with the error:Bug Severity
See examples in the documentation
SEV-2 Major: QuickSight will only contain notifications data older than July 16.
To Reproduce
View the nightly failures in production at https://ca-central-1.quicksight.aws.amazon.com/sn/data-sets/notifications/view
Expected behavior
Dataset created with up to date data.
Impact
CDS can no longer rely on QuickSight for data analytics related to new notifications.
Additional context
See this Slack thread.
TL;DR the joins of the full notification_history table with services, templates, etc are causing the RDS instances to run out of local storage.
Options for fixing: