Handle cloudtrail message where the message is not json

[hamstah]

Stacktrace

2019-09-24T15:42:47.206Z ERROR [CloudtrailSNSNotificationParser] Parsing exception.
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'CloudTrail': was expecting ('true', 'false' or 'null')
 at [Source: CloudTrail validation message.; line: 1, column: 11]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2839) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1903) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2858) ~[graylog.jar:?]
        at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSNSNotificationParser.parse(CloudtrailSNSNotificationParser.java:36) [graylog-plugin-aws-3.1.2.jar:?]
        at org.graylog.aws.inputs.cloudtrail.notifications.CloudtrailSQSClient.getNotifications(CloudtrailSQSClient.java:55) [graylog-plugin-aws-3.1.2.jar:?]
        at org.graylog.aws.inputs.cloudtrail.CloudTrailSubscriber.run(CloudTrailSubscriber.java:89) [graylog-plugin-aws-3.1.2.jar:?]

This is caused by those messages being put in the SQS queue by SNS sometimes (not sure what causes it). The Message field is not JSON so it fails to be parsed and the message stays in the queue and gets refetched forever in a loop, polluting the graylog logs with the stacktrace.

{
  "Type" : "Notification",
  "MessageId" : "xxxxxx",
  "TopicArn" : "arn:aws:sns:us-east-1:xxxxxxxx:cloudtrail-logs-delivery-logs",
  "Message" : "CloudTrail validation message.",
  "Timestamp" : "2019-09-24T14:51:30.832Z",
  "SignatureVersion" : "1",
  "Signature" : "xxxxxx",
  "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-6aad65c2f9911b05cd53efda11f913f9.pem",
  "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:xxxxx:cloudtrail-logs-delivery-logs:xxxx"
}

[hamstah]

Just found https://github.com/Graylog2/graylog-plugin-aws/issues/117 but I checked and we have raw delivery disabled.

[danotorrey]

@hamstah Thanks for the info. I will do some investigation to see if I can figure out why these messages are intermittently appearing in the queue.

[danotorrey]

Hi @hamstah, I really appreciate you bringing this to our attention. These "CloudTrail validation message" SNS notifications get created when the SNS topic for a trail is updated or created. We will need to change the Graylog SNS processing logic to safely ignore them.

Once you have everything set up, these messages should not continue to be generated.

The workaround is to manually delete the validation messages on the SQS queue. There is a View/Delete Messages option in the menu on the main SQS page.

[hamstah]

Hey Dan,

Yeah that's what I've been doing to purge the queue, good to know it's only on update and not periodical at least.

Thanks for looking into it

On Mon, 30 Sep 2019, 8:25 pm Dan Torrey, notifications@github.com wrote:

Hi @hamstah https://github.com/hamstah, I really appreciate you bringing this to our attention. These "CloudTrail validation message" SNS notifications get created when the SNS topic for a trail is updated or created. We will need to change the Graylog SNS processing logic to safely ignore them.

Once you have everything set up, these messages should not continue to be generated.

The workaround is to manually delete the validation messages on the SQS queue. There is a View/Delete Messages option in the menu on the main SQS page.

[image: image] https://user-images.githubusercontent.com/3423655/65905231-cf532900-e385-11e9-8fae-b26c5e6157da.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Graylog2/graylog-plugin-aws/issues/249?email_source=notifications&email_token=AABN7DVPWAWPNH3SNWCDCITQMJADHA5CNFSM4I2BQNPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD76TVII#issuecomment-536689313, or mute the thread https://github.com/notifications/unsubscribe-auth/AABN7DXDH73R5HZ4TT4WFG3QMJADHANCNFSM4I2BQNPA .