Rate exceeded error

[andrewlarioza]

Hi,

This plugin is very helpful thanks. Can you help me with the errors I'm seeing please? I got the following errors on the logs:

{:timestamp=>"2016-08-09T02:59:05.946000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::CloudWatch_Logs log_group=>\"XXX-XXX_server_log\", sincedbpath=>\"/opt/logstash/sincedb/.sincedb\", region=>\"us-east-1\", codec=><LogStash::Codecs::Plain charset=>\"UTF-8\">, interval=>60>\n Error: Rate exceeded", :level=>:error} {:timestamp=>"2016-08-09T02:59:13.045000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::CloudWatch_Logs log_group=>\"XXX-XXX_server_log\", sincedbpath=>\"/opt/logstash/sincedb/.sincedb\", region=>\"us-east-1\", codec=><LogStash::Codecs::Plain charset=>\"UTF-8\">, interval=>60>\n Error: Rate exceeded", :level=>:error}

Thanks!

[lukewaite]

@andrewlarioza It looks like you're hitting #1 which I thought had been resolved by underlying API limit updates to the aws-sdk since I'd not been able to reproduce, but you must be dealing with vastly larger data volumes than I am.

I'll need to look into properly handling rate limiting within the plugin as a longer term solution.

For the short term, if it is an option for you you could create a .sincedb file specifying (epoch microseconds timestamp format) a more recent timeframe with less to ingest.

If you'd like to take a crack at it, I'm open to PRs to fix rate limiting, however I probably won't have the free time in my personal life to look into this until late September, as I am getting married and then will be on vacation.

[andrewlarioza]

Hi Luke,

Congratulations! I don't want to bother you on your upcoming wedding :). Sorry for not informing you later that this was already resolved. But what we're having an issue right now is the Memory leak, It's constantly consuming 10GB of memory in just a span of 30 mins. I would appreciate if you guys can provide a sample config to fix the issue. This is whats on my logstash config right now:

INPUTS: this is the same for 9 cloudwatch log groups

input { cloudwatch_logs { log_group => "tomcatxxx_server_log" sincedb_path => "/dev/null" interval=> "300" region => "us-east-1" } }

OUTPUT:

output{ elasticsearch { hosts => "search-es-domain.us-east-1.es.amazonaws.com:80" ssl => false flush_size => 700 } }

Thanks!

[lukewaite]

@andrewlarioza I think part of your issue here may be that the sincedb_path is set to /dev/null. This is where the plugin keeps track of the last timestamp ingested, so if it's being set to /dev/null, I'm not exactly sure what the behaviour would be, but I think it would likely be starting from the beginning each time.

[jdnurmi]

FWIW, I also hit the rate-exceeded issue when enumerating a very large number of logstreams within a group.

Adding a small sleep in those cases seemed to clear up that issue - see attached for a patch

patch.txt

[dayglojesus]

I'm seeing this issue as well after grafting the patch from issue #7 into my fork. Trying out the patch above as a remedy...

[lukewaite]

Should be resolved now, after the merge of #9