search

logstash-plugins / logstash-output-sqs

Apache License 2.0
4 stars 22 forks source link

Add new multiplex option to logstash-output-sqs plugin #38

Open shalottnet opened 2 years ago

shalottnet commented 2 years ago

Add a new option to allow multiplexing multiple Logstash event messages inside each SQS message. The logstash-input-sqs plugin already has support (via the standard 'json' codec) for demuxing such messages.

This plugin currently supports the use of SQS SendMessageBatch API; but SQS imposes a hard limit of ten messages per API call. By multiplexing instead, we can send potentially hundreds of messages per API call (depending on messages size). In local testing, I saw a 12x reduction in SQS API calls (corresponding to a 12x decrease in SQS costs on the AWS bill).

Tests, docs, etc still WIP; wanted to solicit feedback before going any further.

Thanks.

cla-checker-service[bot] commented 2 years ago

💚 CLA has been signed

shalottnet commented 2 years ago

CLA should be signed now.

shalottnet commented 1 year ago

Any comments or questions on this PR?

CLA was signed, and the Travis CI failures look unrelated.

Would love to get this merged...

Thanks.