amazon-archives / cloudwatch-logs-subscription-consumer

A specialized Amazon Kinesis stream reader (based on the Amazon Kinesis Connector Library) that can help you deliver data from Amazon CloudWatch Logs to any other system in near real-time using a CloudWatch Logs Subscription Filter.
Other
397 stars 152 forks source link

Record failed with message: UnavailableShardsException #20

Open toadkicker opened 7 years ago

toadkicker commented 7 years ago

It seems I'm not able to index logs from cloudwatch:

Record failed: {"index":"cwl-2016.12.19","type":"logging","source":"{\"timestamp\":\"10.0.0.107\",\"@log_stream\":\"production.10.0.1.87.cows.http_access\",\"@timestamp\":1482185012343,\"@message\":\"10.0.0.107 - - [19/Dec/2016:22:03:32 +0000] \\\"HEAD /health HTTP/1.1\\\" 200 - \\\"-\\\" \\\"lua-resty-http/0.08 (Lua) ngx_lua/10005\\\"\",\"request_id\":\"-\",\"event\":\"- [19/Dec/2016:22:03:32 +0000] \\\"HEAD /health HTTP/1.1\\\" 200 - \\\"-\\\" \\\"lua-resty-http/0.08 (Lua) ngx_lua/10005\\\"\",\"@id\":\"33053830297342209648292451756418042022894691916625608704\",\"@log_group\":\"logging\",\"@owner\":\"412642013128\"}","id":"33053830297342209648292451756418042022894691916625608704","version":null,"ttl":null,"create":true}
2016-12-19 22:01:36,404 ERROR ElasticsearchEmitter - Record failed with message: UnavailableShardsException[[cwl-2016.12.19][7] Not enough active copies to meet write consistency of [QUORUM] (have 1, needed 2). Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@7ac0d0eb]
2016-12-19 22:01:36,404 INFO  ElasticsearchEmitter - Emitted 0 records to Elasticsearch
2016-12-19 22:01:36,405 WARN  ElasticsearchEmitter - Cluster health is YELLOW.
2016-12-19 22:01:36,405 WARN  ElasticsearchEmitter - Returning 86 records as failed

I suspect this is simply not understanding the proper configuration for Kibana 4, ES, and using a Cloudwatch logging group. Here's my params:

screen shot 2016-12-19 at 2 12 50 pm