search

amazon-archives / cloudwatch-logs-subscription-consumer

A specialized Amazon Kinesis stream reader (based on the Amazon Kinesis Connector Library) that can help you deliver data from Amazon CloudWatch Logs to any other system in near real-time using a CloudWatch Logs Subscription Filter.
Other
397 stars 152 forks source link

Fails to send JSON with a string prefix and no extracted fields to elasticsearch #10

Open gsterin opened 8 years ago

gsterin commented 8 years ago

There was a change here https://github.com/awslabs/cloudwatch-logs-subscription-consumer/commit/9eef3cd025d8725e62c15b72dc80c36417ccf3c8 to support adding JSON extracted fields with a non-JSON prefix. However, it misses the case where there are no extracted fields but still a string with a JSON prefix: https://github.com/awslabs/cloudwatch-logs-subscription-consumer/blob/master/src/main/java/com/amazonaws/services/logs/connectors/elasticsearch/CloudWatchLogsElasticsearchDocument.java#L100

For example, putting a log event with a message:

test json: {"hello": "world"}

Fails with the following exception in /var/log/cloudwatch-logs-subscription-consumer.log

2015-12-23 18:47:39,825 ERROR KinesisConnectorRecordProcessor - Failed to transform record com.amazonaws.services.logs.subscriptions.CloudWatchLogsEvent@732d486e to output type
java.io.IOException: Error serializing the Elasticsearch document to JSON: A JSONObject text must begin with '{' at 1 [character 2 line 1]
        at com.amazonaws.services.logs.connectors.elasticsearch.ElasticsearchTransformer.fromClass(ElasticsearchTransformer.java:65)
        at com.amazonaws.services.logs.connectors.elasticsearch.ElasticsearchTransformer.fromClass(ElasticsearchTransformer.java:33)
        at com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor.transformToOutput(KinesisConnectorRecordProcessor.java:150)
        at com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor.processRecords(KinesisConnectorRecordProcessor.java:135)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.V1ToV2RecordProcessorAdapter.processRecords(V1ToV2RecordProcessorAdapter.java:42)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:172)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:48)
        at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:23)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.amazonaws.util.json.JSONException: A JSONObject text must begin with '{' at 1 [character 2 line 1]
        at com.amazonaws.util.json.JSONTokener.syntaxError(JSONTokener.java:422)
        at com.amazonaws.util.json.JSONObject.<init>(JSONObject.java:196)
        at com.amazonaws.util.json.JSONObject.<init>(JSONObject.java:323)
        at com.amazonaws.services.logs.connectors.elasticsearch.CloudWatchLogsElasticsearchDocument.getFields(CloudWatchLogsElasticsearchDocument.java:101)
        at com.amazonaws.services.logs.connectors.elasticsearch.CloudWatchLogsElasticsearchDocument.<init>(CloudWatchLogsElasticsearchDocument.java:43)
        at com.amazonaws.services.logs.connectors.elasticsearch.ElasticsearchTransformer.fromClass(ElasticsearchTransformer.java:47)
        ... 11 more
gsterin commented 8 years ago

This line https://github.com/awslabs/cloudwatch-logs-subscription-consumer/blob/master/src/main/java/com/amazonaws/services/logs/connectors/elasticsearch/CloudWatchLogsElasticsearchDocument.java#L101 should be returning new JSONObject(ElasticsearchTransformerUtils.extractJson(message));