search

digitc1 / AWSLandingZone

Repository for AWSLandingZone module developed by DIGIT.C.1
Apache License 2.0
2 stars 0 forks source link

Email subscriptions on SNS topic EC-Landing-Zone-Security-Notification disabled by AWS #136

Closed neisije closed 3 years ago

neisije commented 3 years ago

From: BOTA Florin Mircea (DIGIT) Florin-Mircea.BOTA@ec.europa.eu Sent: Friday, June 4, 2021 08:30 To: HOUMANN Claus (DIGIT) Claus.HOUMANN@ec.europa.eu Cc: PETROVICI Andrei (DIGIT) Andrei.PETROVICI@ec.europa.eu; SIMA Adrian (DIGIT) Adrian.SIMA@ec.europa.eu Subject: FW: Service Issue Affecting AWS Account 127471915879

FYI - it seems the current solution is not a good one in case of large volumes of traffic and multiple alerts. Not sure which alerts are sent from arn:aws:sns:eu-west-1:127471915879:EC-Landing-Zone-Security-Notification and which email address receives them, but it seems that in case of large number of alerts they will no longer be delivered... The account corresponds to the management account (used as SECLOG for all our AWS accounts of D1).

Best regards, Florin.

neisije commented 3 years ago

From: Amazon Web Services, Inc. no-reply-aws@amazon.com Sent: Friday, June 4, 2021 2:39 AM To: DIGIT CLOUD TECH ACCOUNT DIGIT-CLOUD-TECH-ACCOUNT@ec.europa.eu Cc: DIGIT NEXTEUROPA DEVOPS DIGIT-NEXTEUROPA-DEVOPS@ec.europa.eu; willhg@amazon.com; yuriypr@amazon.lu Subject: Service Issue Affecting AWS Account 127471915879

Dear AWS Customer,

This is an automatic notification from the Amazon SNS team.

We have detected a high rate of messages being published to an Amazon SNS topic to which you have email endpoints subscribed. This has resulted in a high volume of messages being sent to the same email addresses, via your Amazon SNS topic. High email send rates to the same destination email addresses can cause external Internet Service Providers (ISPs) to identify sender email addresses, and their associated Internet Protocol (IP) addresses, as sources of email spam. ISPs will often blacklist these email and IP addresses and prevent subsequent emails from being successfully delivered. To avoid blacklisting, we have disabled email subscriptions on this Amazon SNS topic.

Amazon SNS topic ARN: arn:aws:sns:eu-west-1:127471915879:EC-Landing-Zone-Security-Notification Event time: Fri, 4 Jun 2021 00:38:34 GMT

The Amazon SNS team recommends a maximum delivery rate of 10 messages per second to Amazon SNS topics with subscribed email endpoints. If you require a higher delivery rate, we suggest subscribing high-throughput endpoints to your Amazon SNS topic instead, such as Amazon SQS queues, AWS Lambda functions, or HTTP endpoints.

For more information on Amazon SNS service limits, please refer to the AWS Service Limits documentation: https://urldefense.com/v3/__https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html*limits_sns__;Iw!!DOxrgLBm!TclFcZ0nBPZ5eLKI_cRtStcSoP_h07FUMRh68fqu2NpV2rl7kEfLS-Eh_5cqFKmX6zZ1ARPU5upPHetw$

You can restore your email subscriptions by re-subscribing the disabled endpoints to the Amazon SNS topic. Before re-subscribing, however, please ensure that messages are being published to your Amazon SNS topic at a rate less than 10 messages per second.

If you have any questions or concerns, please feel free to reach out to AWS Support: https://urldefense.com/v3/__https://console.aws.amazon.com/support/home__;!!DOxrgLBm!TclFcZ0nBPZ5eLKI_cRtStcSoP_h07FUMRh68fqu2NpV2rl7kEfLS-Eh_5cqFKmX6zZ1ARPU5kNGTzVK$

Thank you, Amazon SNS Team

neisije commented 3 years ago

Design a quickfix to solve this issue. A solution based on a SQS and an (hourly) lambda cron based sending a DIGEST of the notifications pulled from the SQS queue into the SNS topic ?

neisije commented 3 years ago

also evaluate the solution developed by ARHS for DIGIT.C2 which is also based on a DIGEST but trying to send the result via SES to EC email adresses

They are currently struggling to allow SES to send emails to EC.

Their use case is: We’d like to send emails from an application running in AWS. We’d like to send emails from a commission email address, and to other commission email addresses.

What they tried: Setup SES in amazon to send emails from DIGIT-ISHS-CLOUD@ec.europa.eu.

The problem: Emails are sent but not received by commission mailboxes. They are received if the destination is an external email. The commission might be filtering emails because they do not comply with SPF and/or DMARC and/or DKIM.

neisije commented 3 years ago

@silavjy @laurentleonard

Analysis : the source of the notification spam are the notifications generated by the CIS compliancy checks.

My proposal to fix this issue is:

  1. Quickfix : adapt the code of the LZ to reconfigure the cloudwatch event rule 'SECLZ-Config-Rule-Compliance-Change-CloudWatch-Rule' with "CloudWatch log group | config" as a single target => remove the SNS topic from the targets of this event rule

This can be done by adapting the code in CFN/EC-lz-config-cloudtrail-logging.yml and adapting the resource 'ConfigRuleComplianceChangeEventSecLog'

  1. Long Term

Globally reconfigure the Landing Zone to send every x hours a digest of all the notifications produced by the LZ to the SNS topic

neisije commented 3 years ago

Fix of CFN/EC-lz-config-cloudtrail-logging.yml

  # Enable notifications for AWS Config Rule compliance changes
  ConfigRuleComplianceChangeEventSecLog:
    Type: AWS::Events::Rule
    Condition: IsSecLogMasterAccount
    Properties:
      Name: SECLZ-Config-Rule-Compliance-Change-CloudWatch-Rule
      Description: 'Landing Zone rule to send notification on Config Rule compliance changes.'
      EventPattern:
        {
          "source": [
            "aws.config"
          ],
          "detail-type": [
            "Config Rules Compliance Change"
          ]
        }
      State: ENABLED
      Targets:
      - Id: "AwsConfigCloudWatch-Seclog"
        Arn: !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${AwsConfigLogGroup}"
neisije commented 3 years ago

@silavjy @laurentleonard

Summary of what was agreed this morning between us for the quick fix of this issue:

Regarding the 14 Cloudwatch Alarms configured with an action to send the notification via the SNS topic

In case of no side effect detected, the proposal would be:

  1. Reconfigure the 14 Cloudwatch Alarms without actions
  2. Alternatively configure a new Cloudwatch Event Rule to send any event from a cloudwatch Alarm to a cloudwatch log group named /aws/events/alarms (default name)

In consequence, the SNS topic would become useless (unused by the LZ)

neisije commented 3 years ago

Cloudwatch Event Pattern

{ "source": [ "aws.cloudwatch" ], "detail-type": [ "CloudWatch Alarm State Change" ] }

neisije commented 3 years ago

@silavjy @laurentleonard

Based on the feedback recieved from AWS (CASE 8464678521) the proposal to improve the LZ is now the following:

Thoughts ?

neisije commented 3 years ago

For the records, the answer of AWS support to CASE 8464678521:

_

I understand from your case description that you have enabled the monitoring related Security Hub CIS controls (3.1 to 3.14) and have created the required CloudWatch alarms for the same (without any actions currently). You are looking to know if notification action via SNS is required for these controls to pass, or if Security Hub also allows a CloudWatch event rule configured to send the alarm state changes to a log group instead. Please do correct me if I've missed or misunderstood anything.

To run all the CIS 3.1 to 3.14 checks, Security Hub uses custom logic to perform the exact audit steps prescribed for these controls here[1]. The audit steps include checking that a notification via an SNS topic is configured as the CloudWatch alarm's action. If the specified action does not exist or if any of the audit steps do not provide the expected results, then the control fails. (Security Hub does not perform any other checks apart from the specified steps to evaluate the controls.) Therefore currently, since the notification action has not been configured for the CloudWatch alarms, the controls are failing.

If the specified action (notifications via the SNS topic) does not suit your use-case, the controls can be disabled to avoid failed controls. However, you can definitely use a different approach to configure AWS to assist with monitoring and responding to account activities. As you rightly mentioned, configuring a CloudWatch Events rule to send the alarms' state changes to a CloudWatch log group would be a good way to proceed with this. (Please note that this approach will not assist with passing the CIS 3.1-3.14 controls.)

Hope this helps :) In case you face any other issues/queries or if you'd like further clarification on the above, please feel free to reach back and I'll be glad to help.

_

neisije commented 3 years ago

To disable CIS 3.1-3.14 : implement the fix described in https://github.com/digitc1/AWSLandingZone/issues/52

neisije commented 3 years ago

Description of the fix implemented in branch feature/issue136:

neisije commented 3 years ago

Will be fixed by release 1.5.0