search

cds-snc / notification-terraform

Terraform for notification.canada.ca
MIT License
13 stars 6 forks source link

2.13.3 #1429

Closed ben851 closed 3 months ago

ben851 commented 3 months ago

Summary | Résumé

https://github.com/cds-snc/notification-terraform/pull/1409 - New Relic Terraform Alert @P0NDER0SA https://github.com/cds-snc/notification-terraform/pull/1412 - Separating Staging TF Workflows Into Separate Jobs @ben851 https://github.com/cds-snc/notification-terraform/pull/1417 - K8s Worker UPdates @ben851 https://github.com/cds-snc/notification-terraform/pull/1420 - Git checkout added to bump version @ben851 https://github.com/cds-snc/notification-terraform/pull/1418 - Adding filter to workflows for newrelic @ben851 https://github.com/cds-snc/notification-terraform/pull/1421 - OIDC In staging Take 2 @ben851 https://github.com/cds-snc/notification-terraform/pull/1415 - 2 Dashboard Query Fix https://github.com/cds-snc/notification-terraform/pull/1419 - Production TF Plan/Apply split into multiple jobs @ben851

Related Issues | Cartes liées

Release

Test instructions | Instructions pour tester la modification

Release Instructions | Instructions pour le déploiement

None.

Reviewer checklist | Liste de vérification du réviseur

⚠ If boxes cannot be checked off before merging the PR, they should be moved to the "Release Instructions" section with appropriate steps required to verify before release. For example, changes to celery code may require tests on staging to verify that performance has not been affected.

github-actions[bot] commented 3 months ago

Production: newrelic

✅   Terraform Init: success ✅   Terraform Validate: success ✅   Terraform Format: success ✅   Terraform Plan: success

Plan: 1 to add, 0 to change, 0 to destroy
Show summary | CHANGE | NAME | |--------|--------------------------------------------------------------------------------------| | add | `newrelic_nrql_alert_condition.tf_lambda_api_errors_count_anomaly_unexpected_errors` |
Show plan ```terraform Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # newrelic_nrql_alert_condition.tf_lambda_api_errors_count_anomaly_unexpected_errors will be created + resource "newrelic_nrql_alert_condition" "tf_lambda_api_errors_count_anomaly_unexpected_errors" { + account_id = 2691974 + aggregation_delay = "300" + aggregation_method = "event_flow" + aggregation_window = 60 + baseline_direction = "upper_and_lower" + enabled = true + entity_guid = (known after apply) + id = (known after apply) + name = "Staging - Terraform - [Lambda API] Errors count anomaly (Unexpected Errors)" + policy_id = 2801728 + type = "baseline" + violation_time_limit = (known after apply) + violation_time_limit_seconds = 86400 + critical { + operator = "above" + threshold = 6 + threshold_duration = 300 + threshold_occurrences = "all" } + nrql { + query = "SELECT count(*) FROM AwsLambdaInvocationError WHERE (`entityGuid`='MjY5MTk3NHxJTkZSQXxOQXwtNzgwNDUyNTc5NzAyODI1NTcyNw') and error.class NOT IN ('app.v2.errors:BadRequestError','jsonschema.exceptions:ValidationError', 'sqlalchemy.exc:NoResultFound', 'app.authentication.auth:AuthError', 'werkzeug.exceptions:MethodNotAllowed') and error.message NOT LIKE '{\\'result\\': \\'error\\', \\'message\\': {\\'password\\': [\\'Incorrect password\\']}}'" } + warning { + operator = "above" + threshold = 3 + threshold_duration = 300 + threshold_occurrences = "all" } } Plan: 1 to add, 0 to change, 0 to destroy. ───────────────────────────────────────────────────────────────────────────── Saved the plan to: plan.tfplan To perform exactly these actions, run the following command to apply: terraform apply "plan.tfplan" ```
github-actions[bot] commented 3 months ago

Production: common

✅   Terraform Init: success ✅   Terraform Validate: success ✅   Terraform Format: success ✅   Terraform Plan: success ✅   Conftest: success 

Plan: 0 to add, 1 to change, 0 to destroy
Show summary | CHANGE | NAME | |--------|---------------------------------------------------------------------------------| | update | `aws_cloudwatch_metric_alarm.sns-sms-success-rate-canadian-numbers-critical[0]` |
Show plan ```terraform Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # aws_cloudwatch_metric_alarm.sns-sms-success-rate-canadian-numbers-critical[0] will be updated in-place ~ resource "aws_cloudwatch_metric_alarm" "sns-sms-success-rate-canadian-numbers-critical" { ~ actions_enabled = false -> true id = "sns-sms-success-rate-canadian-numbers-critical" tags = {} # (17 unchanged attributes hidden) } Plan: 0 to add, 1 to change, 0 to destroy. Warning: Argument is deprecated with aws_s3_bucket.csv_bucket, on s3.tf line 5, in resource "aws_s3_bucket" "csv_bucket": 5: resource "aws_s3_bucket" "csv_bucket" { Use the aws_s3_bucket_server_side_encryption_configuration resource instead (and 71 more similar warnings elsewhere) ───────────────────────────────────────────────────────────────────────────── Saved the plan to: plan.tfplan To perform exactly these actions, run the following command to apply: terraform apply "plan.tfplan" ```
Show Conftest results ```sh WARN - plan.json - main - Missing Common Tags: ["aws_athena_workgroup.ad_hoc"] WARN - plan.json - main - Missing Common Tags: ["aws_athena_workgroup.build_tables"] WARN - plan.json - main - Missing Common Tags: ["aws_athena_workgroup.primary"] WARN - plan.json - main - Missing Common Tags: ["aws_athena_workgroup.support"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.aws_health[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.route53_resolver_query_log[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.sns_deliveries[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.sns_deliveries_failures[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.sns_deliveries_failures_us_west_2[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.sns_deliveries_us_west_2[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-bulk-not-being-processed-critical[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-bulk-not-being-processed-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-inflights-not-being-processed-critical[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-inflights-not-being-processed-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-not-being-processed-critical[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-not-being-processed-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.contact-3-500-error-15-minutes-critical[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.document-download-bucket-size-warning[0]"] WARN - plan.json - main - Missing Common Tags:... ```
github-actions[bot] commented 3 months ago

Production: pinpoint_to_sqs_sms_callbacks

✅   Terraform Init: success ✅   Terraform Validate: success ✅   Terraform Format: success ✅   Terraform Plan: success ✅   Conftest: success 

Plan: 0 to add, 1 to change, 0 to destroy
Show summary | CHANGE | NAME | |--------|----------------------------------------| | update | `aws_cloudwatch_dashboard.pinpoint[0]` |
Show plan ```terraform Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # aws_cloudwatch_dashboard.pinpoint[0] will be updated in-place ~ resource "aws_cloudwatch_dashboard" "pinpoint" { ~ dashboard_body = jsonencode( ~ { ~ widgets = [ # (12 unchanged elements hidden) { height = 9 properties = { markdown = <<-EOT ## Message flow After an SMS has been sent by Pinpoint, the delivery details are stored in CloudWatch Log groups: - [sns/ca-central-1/296255494825/PinpointDirectPublishToPhoneNumber](#logsV2:log-groups/log-group/sns$252Fca-central-1$252F296255494825$252FPinpointDirectPublishToPhoneNumber) for successful deliveries - [sns/ca-central-1/296255494825/PinpointDirectPublishToPhoneNumber/Failure](#logsV2:log-groups/log-group/sns$252Fca-central-1$252F296255494825$252FPinpointDirectPublishToPhoneNumber$252FFailure) for failures The log groups are subscribed the Lambda function [pinpoint-to-sqs-sms-callbacks](#/functions/pinpoint-to-sqs-sms-callbacks?tab=configuration). This Lambda adds messages to the SQS queue `delivery-receipts` to trigger the Celery task in charge of updating notifications in the database, `process-pinpoint-result`. See the relevant [AWS documentation](https://docs.aws.amazon.com/sns/latest/dg/sms_stats_cloudwatch.html#sns-viewing-cloudwatch-logs) for these messages. EOT } type = "text" width = 6 x = 18 y = 39 }, ~ { ~ properties = { ~ query = <<-EOT - SOURCE 'sns/ca-central-1/296255494825/PinpointDirectPublishToPhoneNumber/Failure' | fields @timestamp as Timestamp, notification.messageId as MessageID, status, delivery.destination as Destination, delivery.providerResponse as ProviderResponse + SOURCE 'sns/ca-central-1/296255494825/PinpointDirectPublishToPhoneNumber/Failure' | fields @timestamp as Timestamp, messageId as MessageID, messageStatus as status, destinationPhoneNumber as Destination, messageStatusDescription as ProviderResponse | sort @timestamp desc | limit 20 EOT # (4 unchanged attributes hidden) } # (5 unchanged attributes hidden) }, { height = 6 properties = { annotations = { horizontal = [ { fill = "above" label = "Above 1 minute" value = 60 }, { fill = "above" label = "Above 30 seconds" value = 30 }, ] } end = "P0D" metrics = [ [ "NotificationCanadaCa", "production_notifications_celery_clients_sns_request-time", "metric_type", "timing", { visible = false }, ], [ ".", "production_notifications_celery_sms_total-time", ".", ".", ], ] period = 60 region = "ca-central-1" setPeriodToTimeRange = true sparkline = true stacked = false start = "-PT3H" stat = "p90" title = "SMS (SNS and Pinpoint) sending time in seconds" view = "timeSeries" yAxis = { left = { label = "Sending time (sent_at - created_at)" min = 0 showUnits = false } } } type = "metric" width = 9 x = 9 y = 5 }, ~ { ~ properties = { ~ query = <<-EOT - SOURCE 'sns/ca-central-1/296255494825/DirectPublishToPhoneNumber/Failure' | SOURCE 'sns/ca-central-1/296255494825/PinpointDirectPublishToPhoneNumber' | stats avg(delivery.dwellTimeMsUntilDeviceAck / 1000 / 60) as Avg_carrier_time_minutes, count(*) as Number by delivery.phoneCarrier as Carrier + SOURCE 'sns/ca-central-1/296255494825/PinpointDirectPublishToPhoneNumber' | filter isFinal = 1 + | stats avg((eventTimestamp - messageRequestTimestamp) / 1000 / 60) as Avg_carrier_time_minutes, count(*) as Number by carrierName as Carrier EOT # (3 unchanged attributes hidden) } # (5 unchanged attributes hidden) }, { height = 6 properties = { query = "SOURCE 'sns/ca-central-1/296255494825/PinpointDirectPublishToPhoneNumber' | SOURCE 'sns/ca-central-1/296255494825/DirectPublishToPhoneNumber/Failure' | stats avg(delivery.dwellTimeMsUntilDeviceAck / 1000 / 60) as Avg_carrier_time_minutes by bin(30s)" region = "ca-central-1" stacked = false title = "dwellTimeMsUntilDeviceAck" view = "timeSeries" } type = "log" width = 24 x = 0 y = 57 }, # (7 unchanged elements hidden) ] } ) id = "SMS-Pinpoint" # (2 unchanged attributes hidden) } Plan: 0 to add, 1 to change, 0 to destroy. ───────────────────────────────────────────────────────────────────────────── Saved the plan to: plan.tfplan To perform exactly these actions, run the following command to apply: terraform apply "plan.tfplan" ```
Show Conftest results ```sh WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.pinpoint_deliveries"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.pinpoint_deliveries_failures"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.pinpoint_to_sqs_sms_callbacks_log_group[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.lambda-image-pinpoint-delivery-receipts-errors-critical[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.lambda-image-pinpoint-delivery-receipts-errors-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.logs-1-500-error-1-minute-warning-pinpoint_to_sqs_sms_callbacks-api[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.logs-10-500-error-5-minutes-critical-pinpoint_to_sqs_sms_callbacks-api[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.pinpoint-sms-blocked-as-spam-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.pinpoint-sms-phone-carrier-unavailable-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.pinpoint-sms-rate-exceeded-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.pinpoint-sms-success-rate-critical[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.pinpoint-sms-success-rate-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.total-sms-spending-critical[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.total-sms-spending-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.pinpoint_logs"] WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.pinpoint_logs"] 35 tests, 19 passed, 16 warnings, 0 failures, 0 exceptions ```
github-actions[bot] commented 3 months ago

Production: eks

✅   Terraform Init: success ✅   Terraform Validate: success ✅   Terraform Format: success ✅   Terraform Plan: success ✅   Conftest: success

⚠️   Warning: resources will be destroyed by this change!

Plan: 1 to add, 1 to change, 1 to destroy
Show summary | CHANGE | NAME | |----------|----------------------------------------------------------------| | update | `aws_eks_node_group.notification-canada-ca-eks-node-group-k8s` | | recreate | `aws_secretsmanager_secret_version.eks_karpenter_ami_id` |
Show plan ```terraform Resource actions are indicated with the following symbols: ~ update in-place -/+ destroy and then create replacement Terraform will perform the following actions: # aws_eks_node_group.notification-canada-ca-eks-node-group-k8s will be updated in-place ~ resource "aws_eks_node_group" "notification-canada-ca-eks-node-group-k8s" { id = "notification-canada-ca-production-eks-cluster:notification-canada-ca-production-eks-primary-node-group-k8s" ~ release_version = "1.30.0-20240605" -> "1.30.0-20240615" tags = { "CostCenter" = "notification-canada-ca-production" "Name" = "notification-canada-ca" "karpenter.sh/discovery" = "notification-canada-ca-production-eks-cluster" } # (15 unchanged attributes hidden) # (3 unchanged blocks hidden) } # aws_secretsmanager_secret_version.eks_karpenter_ami_id must be replaced -/+ resource "aws_secretsmanager_secret_version" "eks_karpenter_ami_id" { ~ arn = "arn:aws:secretsmanager:ca-central-1:296255494825:secret:EKS_KARPENTER_AMI_ID-ApW79n" -> (known after apply) ~ id = "arn:aws:secretsmanager:ca-central-1:296255494825:secret:EKS_KARPENTER_AMI_ID-ApW79n|terraform-20240626133942357600000004" -> (known after apply) ~ secret_string = (sensitive value) # forces replacement ~ version_id = "terraform-20240626133942357600000004" -> (known after apply) ~ version_stages = [ - "AWSCURRENT", ] -> (known after apply) # (1 unchanged attribute hidden) } Plan: 1 to add, 1 to change, 1 to destroy. ───────────────────────────────────────────────────────────────────────────── Saved the plan to: plan.tfplan To perform exactly these actions, run the following command to apply: terraform apply "plan.tfplan" ```
Show Conftest results ```sh WARN - plan.json - main - Missing Common Tags: ["aws_acm_certificate.client_vpn"] WARN - plan.json - main - Missing Common Tags: ["aws_acm_certificate.notification-canada-ca"] WARN - plan.json - main - Missing Common Tags: ["aws_acm_certificate.notification-canada-ca-alt[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_alb.notification-canada-ca"] WARN - plan.json - main - Missing Common Tags: ["aws_alb_listener.internal_alb_tls"] WARN - plan.json - main - Missing Common Tags: ["aws_alb_listener.notification-canada-ca"] WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.internal_nginx_http"] WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.notification-canada-ca-admin"] WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.notification-canada-ca-api"] WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.notification-canada-ca-document"] WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.notification-canada-ca-document-api"] WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.notification-canada-ca-documentation"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.notification-canada-ca-eks-application-logs[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.notification-canada-ca-eks-cluster-logs[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.notification-canada-ca-eks-prometheus-logs[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.admin-evicted-pods[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.admin-pods-high-cpu-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.admin-pods-high-memory-warning[0]"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.admin-replicas-unavailable[0]"] WARN - plan.json - main - Missing Common Tags:... ```
github-actions[bot] commented 3 months ago

Production: quicksight

✅   Terraform Init: success ✅   Terraform Validate: success ✅   Terraform Format: success ✅   Terraform Plan: success ✅   Conftest: success 

Plan: 0 to add, 1 to change, 0 to destroy
Show summary | CHANGE | NAME | |--------|-------------------------------| | update | `aws_s3_object.manifest_file` |
Show plan ```terraform Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: # aws_s3_object.manifest_file will be updated in-place ~ resource "aws_s3_object" "manifest_file" { ~ etag = "3696c2177cd9e1be28ff597c24b10ae0" -> "221f592f333f2fc284626cfdb8c4bc80" id = "quicksight/s3-manifest-sms-usage.json" tags = {} + version_id = (known after apply) # (12 unchanged attributes hidden) } Plan: 0 to add, 1 to change, 0 to destroy. ───────────────────────────────────────────────────────────────────────────── Saved the plan to: plan.tfplan To perform exactly these actions, run the following command to apply: terraform apply "plan.tfplan" ```
Show Conftest results ```sh WARN - plan.json - main - Missing Common Tags: ["aws_cloudformation_stack.sms-usage-notifications"] WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.quicksight-rds"] WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.quicksight-s3-usage"] WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.quicksight_vpc_connection_ec2"] WARN - plan.json - main - Missing Common Tags: ["aws_iam_policy.quicksight_vpc_connection_iam"] WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.quicksight"] WARN - plan.json - main - Missing Common Tags: ["aws_iam_role.vpc_connection_role"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_data_set.jobs"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_data_set.login_events"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_data_set.notifications"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_data_set.organisation"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_data_set.services"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_data_set.sms_usage"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_data_set.templates"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_data_set.users"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_data_source.rds"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_data_source.s3_sms_usage"] WARN - plan.json - main - Missing Common Tags: ["aws_quicksight_vpc_connection.rds"] WARN - plan.json - main - Missing Common Tags: ["aws_s3_object.manifest_file"] 38 tests, 19 passed, 19 warnings, 0 failures, 0 exceptions ```