Decode Error on message notification

Moridinmhael commented 4 years ago

Getting this error on versions 5:15. 5:16.3

When trying to Publish messages via SNS, AWS logs return 200 result, however, below error appears in Mattermost server logs and no message recieved.

Subscription confirmation works, and done.

AWSSNS HandleNotification {"plugin_id": "com.mattermost.aws-sns"} AWSSNS HandleNotification Decode Error on message notification {"plugin_id": "com.mattermost.aws-sns", "err=": "invalid character 'e' in literal true (expecting 'r')"}

manish-aggarwal commented 4 years ago

I am getting a similar error in version 5.18

Even for me subscription confirmation worked fine, but test messages from SNS to mattermost are not going.

jfrerich commented 4 years ago

Hi @Moridinmhael and @manish-aggarwal. I'll try to help resolve this issue. I have a couple of questions that would help me narrow down the cause of the following error:

AWSSNS HandleNotification Decode Error on message notification {"plugin_id": "com.mattermost.aws-sns", "err=": "invalid character 'e' in literal true (expecting 'r')"}

The error message occurs when the plugin receives a message and tries to unmarshal the data. Specifically, this error is received when unmarshalling the Message field of the json data object.

{
  "Type" : "Notification",
  "MessageId" : "7ec2dc18-24b9-5c2b-83da-8afa46fe1972",
  "TopicArn" : "arn:aws:sns:us-east-2:318162992057:hello-mattermost",
  "Subject" : "ALARM: \"awsec2-i-026ba790669e62dc8-CPU-Utilization\" in US East (Ohio)",
  "Message" : "{\"AlarmName\":\"awsec2-i-026ba790669e62dc8-CPU-Utilization\",\"AlarmDescription\":\"Created from EC2 Console\",\"AWSAccountId\":\"318162992057\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [0.1322404371585012 (11/03/20 03:45:00)] was less than or equal to the threshold (1.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2020-03-11T03:50:31.090+0000\",\"Region\":\"US East (Ohio)\",\"OldStateValue\":\"INSUFFICIENT_DATA\",\"Trigger\":{\"MetricName\":\"CPUUtilization\",\"Namespace\":\"AWS/EC2\",\"StatisticType\":\"Statistic\",\"Statistic\":\"AVERAGE\",\"Unit\":null,\"Dimensions\":[{\"value\":\"i-026ba790669e62dc8\",\"name\":\"InstanceId\"}],\"Period\":60,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"LessThanOrEqualToThreshold\",\"Threshold\":1.0,\"TreatMissingData\":\"\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
  "Timestamp" : "2020-03-11T03:50:31.131Z",
  "SignatureVersion" : "1",
  "Signature" : "jsVPK9gWERnVMfbVuGUsjKBUTT03CDu0diMoEz0Y8zUwS3vutTu6Fzov3I8QtZBww0hQSk9vNrGW5sNAUM0eeqrMLHWpQ++fwBsgT2wtR8yqTYGcfmBftivjjzbW8U+g3Htwv3aT9iEzOcNlsx27pd8eT2aMhlMya5JbpXk5W35LXxkdBWxkkUwIUuPXV0nhtFm1V5vkjztof/3vUr4OeSwmb4eHp9vf3HcH6tkIazn2DKU5Iwwt+roqtYV2lWtu2/S5WVnYYg5sAoaVyFkAj4FGF4IsZOxmewrpgrtmxYQMD/k9rIdLL7VgGOevwn8TpkRTVFMp4mAkB4Apbjg8VQ==",
  "SigningCertURL" : "https://sns.us-east-2.amazonaws.com/SimpleNotificationService-a86cb10b4e1f29c941702d737128f7b6.pem",
  "UnsubscribeURL" : "https://sns.us-east-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-2:318162992057:hello-mattermost:45228a0a-7c6b-4296-998b-d12a292917d4"
}

My suspicion is the json data is malformed (based on the semi-cryptic error message).

What would help me debug would be any of the additional information:

Capture the response body of a failed message by monitoring the network traffic in Mattermost
- look for a POST to /plugins/com.mattermost.aws-sns
OR Capture this request from AWS. (I am not quite sure how, but assume possible)
What type of alarm is being used to send the message
Capture the post history from the alarms page.
- Note the time Date timestamp here will match (very closely) to the debug log statements you are seeing in the mattermost logs.

levb commented 4 years ago

@jfrerich Can you maybe instrument the code so that we dump the payload in the logs on all UnmarshalJSON errors?

jfrerich commented 4 years ago

@Moridinmhael and @manish-aggarwal,

Thank you for the ticket and observation. I have added code that will log the notification, from aws-sns, when an unmarshaling error occurs. https://github.com/mattermost/mattermost-plugin-aws-SNS/pull/33

If you see this error again, I would love to see the logged error. If you find that it is something that is addressed on the aws-sns side, I would still like to document the error and add as a FAQ.

GilDev commented 3 years ago

Version 1.2.0, I’m getting this in the logs when publishing a SNS message from the AWS console:

{"level":"error","ts":1633339827.6983178,"caller":"mlog/sugar.go:25","msg":"AWSSNS HandleNotification Decode Error on RDS-Event message notification","plugin_id":"com.mattermost.aws-sns","err":"invalid character 'C' looking for beginning of value","message":"Ceci est un nouveau test"}
{"level":"error","ts":1633339827.7012866,"caller":"mlog/sugar.go:25","msg":"AWSSNS HandleNotification Decode Error on CloudWatch message notification","plugin_id":"com.mattermost.aws-sns","err":"invalid character 'C' looking for beginning of value","message":"Ceci est un nouveau test"}

GilDev commented 2 years ago

So yeah, only CloudWatch alarms are supported, no raw message.

mattermost-community / mattermost-plugin-aws-SNS

Decode Error on message notification #28