search

amazon-archives / logstash-output-cloudwatchlogs

A logstash plugin that allows to send logs to AWS CloudWatch Logs service.
Other
37 stars 34 forks source link

The given sequenceToken is invalid errors #2

Open vaijab opened 9 years ago

vaijab commented 9 years ago

I am getting errors like one below:

Oct 12 13:37:08 ip-10-50-0-244.eu-west-1.compute.internal docker[4880]: The given sequenceToken is invalid. The next expected sequenceToken is: 49545729770787039968577718761852552028493222699855781714 {:level=>:warn, :exception=>Aws::CloudWatchLogs::Errors::InvalidSequenceTokenException, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.14/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.14/lib/aws-sdk-core/plugins/param_converter.rb:21:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.14/lib/seahorse/client/plugins/response_target.rb:21:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.14/lib/seahorse/client/request.rb:70:in `send_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.14/lib/seahorse/client/base.rb:207:in `put_log_events'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:159:in `put_log_events'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:139:in `flush'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:138:in `flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:102:in `register'", "org/jruby/RubyProc.java:271:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:338:in `deq'", "org/jruby/RubyKernel.java:1511:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:333:in `deq'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-cloudwatchlogs-0.9.0/lib/logstash/outputs/cloudwatchlogs.rb:101:in `register'"]}

I have multiple instances running logstash and pushing to the same LOG_GROUP_NAME/LOG_STREAM_NAME. Would running multiple logstash instances cause an error like the one above?

wanghq commented 9 years ago

Hi @vaijab, yes, that could cause above error. A log stream is sequence of log events from a single emitter of logs. You can read more about CloudWatch Logs concepts at http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchLogs.html#CloudWatchLogsConcepts. Pointing multiple logstash instances to the same log stream could cause data duplicates and reduce the ingestion throughput.

Here is a simple example. If you have several apache servers, each generating access log and error log. Then you could have two log groups (e.g. AccessLog and ErrorLog) and use instance id as log stream name, and you end up with below log streams.

#log group/log stream

AccessLog/i-instance123
AccessLog/i-instance456
AccessLog/i-instance789

ErrorLog/i-instance123
ErrorLog/i-instance456
ErrorLog/i-instance789
vaijab commented 9 years ago

@wanghq thanks for the explanation. I had a feeling that was the case. I'd love to be able to create dynamic streams based on types of events, but see #1 why I cannot do that unfortunately.

Today, in containerized world instances don't mean much. Containers move around and can be short-lived.

teebu commented 7 years ago

i only have 1 output, and im seeing these errors.