search

SumoLogic / sumologic-aws-lambda

A collection of lambda functions to collect data from Cloudwatch, Kinesis, VPC Flow logs, S3, security-hub and AWS Inspector
Other
151 stars 135 forks source link

kinesis firehose processor #87

Open RajanAOz opened 5 years ago

RajanAOz commented 5 years ago

I have installed the kinesis firehose processor but seeing weird characters in SumoLogic.

I have used the readme from the following github url https://github.com/SumoLogic/sumologic-aws-lambda/tree/master/kinesisfirehose-processor to implement the kinesis firehose processor.

When collecting the data from S3 in SumoLogic I see the following result. Please help figuring out what is going on...

�5�[��0�� ��+v|ޢ�ݶ���D�@Q5E"��fQ����ۇy9gt��w#�O��7x� _����sY��˒�I����d���a܈<����2��!9^Cv��1�MW�H�Xӡ#����m��.c�w�&����)ߘߠ�l����GrG�m�������l��*�eY��9�P�S��BsTRjt���"�Ņ��S�ss�~d���O���C"�G���wq�o�9�J!5�\HõB��R0�Ų4`�)%� �R(51�M"�MPxRr�2���?�_Vź����KZ�~\̸���Բ<��:O������0ۤ�/��'�������q$��

bhavikkumar commented 5 years ago

Are you collecting cloudwatch logs? If so then it will be gzipped. Below is a example of how to decode it on bash.

echo -n "H4sIAAAAAAAAAJWRTWsbMRCG/8ueLZjRjL5yc9NNLnZDapemlFAkrTYstb3Lep0Qgv97x00KgTSHnAQzmkeP3nmqtmW/j3dl/TiU6qz6PF/Pfy3r1Wp+WVezqn/YlVHK2pK3Hr0Jxkt5099djv1hkE7uh0eVHzZqE7epiarb3fe/ixzDYVJoELRhssYQqsXLlEJ3jd8//biy4QYWz7jVNJa4/TDveQwV+qsada0v/HnthLg/pH0eu2Hq+t1Ft5nKuK/Ofn4EvnpDUAu7Xi6/LL9en3/z1e1f7fq+7KYT+qnqGrEnsi54AGS2wbHWxjCjoWAYGawmzawByIG3Dp0JzjOxsaI8dbKJKW4l1BcTdgg+zP5tSPCeQ/Bso/I+o+I2kUptjgrRlQyasslUHWdvZRwGJ4+HYJGCtiKgQTYKSJ4gODLgAkpFk3f0rkyA1zLGSsvoVsVCRTFakUkNqKxt1IyFc8T/y0gEmoHZo5a/W9HhU0TeWHMyIJaoQC6zDvC+DL6WSW3MqZSkiolJcWoalWybJSNIJTXcRgjV8fb4BwwLrNzwAgAA" | base64 -d | zcat

alejandroortuno commented 4 years ago

Hi,

I am also sending Guarduty events from a cloudwatch rule event and the data record of kinesis is not compressed making the lambda to fail. Can we make a modification so this is not needed or based on the case for the source event?