search

sampointer / fluent-plugin-cloudwatch-ingest

Alternative to ryotarai/fluent-plugin-cloudwatch-logs for ingesting AWS Cloudwatch logs via fluentd
GNU General Public License v3.0
11 stars 10 forks source link

ThrottlingException handling enhansement #23

Closed akiraimafuji closed 6 years ago

akiraimafuji commented 6 years ago

We cannot completely avoid ThrottlingException because it is mostly matter of concurrency. So I suggest:

log sample

ThrottlingException

state is saved before/after the exception

2017-12-14 11:56:31 +0900 [info]: #0 [test1] processing stream: test-stream
2017-12-14 11:56:31 +0900 [info]: #0 [test1]  log_group_name="test-group" log_stream_name="test-stream" next_token="f/33743856321123405891826378568446772091912687289125109763" limit=10000 start_time=0 start_from_head=true
2017-12-14 11:56:34 +0900 [error]: #0 [test1] Unable to retrieve events for stream test-stream in group test-group: #<Aws::CloudWatchLogs::Errors::ThrottlingException: Rate exceeded>

2017-12-14 11:58:38 +0900 [info]: #0 [test1] processing stream: test-stream
2017-12-14 11:58:38 +0900 [info]: #0 [test1]  log_group_name="test-group" log_stream_name="test-stream" next_token="f/33743856321123405891826378568446772091912687289125109763" limit=10000 start_time=0 start_from_head=true

InvalidParameterException

next_token and start_time parameter changes after the exception, as it was.

2017-12-14 12:13:24 +0900 [info]: #0 [test1] processing stream: test-stream
2017-12-14 12:13:24 +0900 [info]: #0 [test1]  log_group_name="test-group" log_stream_name="test-stream" next_token="f/INVALIDTOKEN" limit=10000 start_time=0 start_from_head=true
2017-12-14 12:13:24 +0900 [error]: #0 [test1] cloudwatch token is expired or broken. trying with timestamp.
2017-12-14 12:13:24 +0900 [info]: #0 [test1]  log_group_name="test-group" log_stream_name="test-stream" next_token=nil limit=10000 start_time=1513126849382 start_from_head=true
2017-12-14 12:13:24 +0900 [info]: #0 [test1] 1 events processed for stream test-stream

2017-12-14 12:21:44 +0900 [info]: #0 [test1] processing stream: test-stream
2017-12-14 12:21:44 +0900 [info]: #0 [test1]  log_group_name="test-group" log_stream_name="test-stream" next_token="f/33743856321123405891826378568446772091912687289125109763" limit=10000 start_time=0 start_from_head=true
2017-12-14 12:21:44 +0900 [info]: #0 [test1] 0 events processed for stream test-stream
sampointer commented 6 years ago

Hey. Nice work.

Would it be possible for you to do the following?

With all of that done I'd be happy to push this as the next 1.7.0 release candidate.

Thanks again.

akiraimafuji commented 6 years ago

@sampointer Thank you for the reply. I modified README.md and added Seahorse::Client::NetworkingError to save state which I experienced 2 times in this month on AWS EC2.

https://github.com/sampointer/fluent-plugin-cloudwatch-ingest/pull/23/files/84961ff8a2c1d734777b53b9f65ea487dee61c3d..ed63ad5360886eeae5dde1a573bb3c70cba98672

If there is any problems please tell me.

sampointer commented 6 years ago

Merged to master but failing the linter on syntax, so I haven't cut a release candidate: https://circleci.com/gh/sampointer/fluent-plugin-cloudwatch-ingest/309

sampointer commented 6 years ago

Pushed as 1.7.0.rc3