search

threadwaste / logstash-codec-cloudwatch_logs

An experimental codec for parsing CloudWatch Logs subscriptions from Kinesis.
Other
16 stars 7 forks source link

plugin seems to assumes "message" is a single value string, not json(Cloudtrail) #3

Closed gregorycox closed 6 years ago

gregorycox commented 7 years ago

What would be involved in the plugin trying to see if "message" is json, read it as json, if not then assume it's a string.

Example of what is happening today, plugin sends below to elasticsearch.

{ "_index": "logstash-2017.05.04", "_type": "logs", "_id": "AVvUlq6og87QN6AvRyp-", "_score": null, "_source": { "owner": "1234", "subscriptionFilters": [ "elk-stack-nonprod-ElkStack-1OKD76401VAYD-KinesisStackSubscribingCloudWatch-1I9QBWJQ1O6DS-SubscriptionFilter-1035A90CORGG4" ], "@timestamp": "2017-05-04T17:51:33.708Z", "messageType": "DATA_MESSAGE", "logGroup": "nonprod", "@version": "1", "logStream": "1234_CloudTrail_us-east-1", "id": "33315535816896139295441340713170715507857824658564710408", "message": "{\"eventVersion\":\"1.05\",\"userIdentity\":{\"type\":\"AssumedRole\",\"principalId\":\"1234:spotinst.session.1493918167758\",\"arn\":\"arn:aws:sts::1234:assumed-role/spotinst-iam-stack-SpotinstRole-7WC3U4XDRF0O/spotinst.session.1493918167758\",\"accountId\":\"1234\",\"accessKeyId\":\"1234\",\"sessionContext\":{\"attributes\":{\"mfaAuthenticated\":\"false\",\"creationDate\":\"2017-05-04T17:16:07Z\"},\"sessionIssuer\":{\"type\":\"Role\",\"principalId\":\"1234\",\"arn\":\"arn:aws:iam::1234:role/spotinst-iam-stack-SpotinstRole-7WC3U4XDRF0O\",\"accountId\":\"1234\",\"userName\":\"spotinst-iam-stack-SpotinstRole-7WC3U4XDRF0O\"}}},\"eventTime\":\"2017-05-04T17:46:13Z\",\"eventSource\":\"ec2.amazonaws.com\",\"eventName\":\"DescribeAccountAttributes\",\"awsRegion\":\"ap-southeast-2\",\"sourceIPAddress\":\"1234\",\"userAgent\":\"aws-sdk-nodejs/2.48.0 linux/v5.11.1\",\"requestParameters\":{\"accountAttributeNameSet\":{\"items\":[{\"attributeName\":\"supported-platforms\"}]},\"filterSet\":{}},\"responseElements\":null,\"requestID\":\"1234\",\"eventID\":\"056b6ccf-986c-41fa-8c3f-5093db589088\",\"eventType\":\"AwsApiCall\",\"recipientAccountId\":\"1234\"}" },

Would appreciate any help, even just a suggestion to work around it, and I can work on that.

Thanks

gregorycox commented 7 years ago

currently showing up as 

"logStream": "123455677_CloudTrail_us-east-1",
"id": "33323333468277649116120603193783649704976278433685176322",
"message": "{\"eventVersion\":\"1.05\",\"userIdentity\":{\"type\":\"AssumedRole\",\"principalId\

The message portion seems to be read as a string when it's JSON????

gregorycox commented 7 years ago

version: logstash-input-kinesis (2.0.3) logstash-codec-cloudwatch_logs (0.0.2)

threadwaste commented 7 years ago

@gregorycox This is the intended behavior. I intentionally left any opinion out of the codec. Even if I think everyone should be using a structured log format. :)

What you can do is add a json filter on the message field. Have a look here:

https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

Let me know if this doesn't work for your needs.