Send events to firehose with AWS SDK and the put_record_batch method.
Events will be sent in batches, following the base multi_receive_encoded method.
If number or size of events exceeds the AWS limits, sub-batches will be created.
The limits can be overriden by the users and they must define at least a delivery_stream_name configuration.
Checklist
[x] My code follows the style guidelines of this project
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] I have made corresponding change to the default configuration files (and/or docker env variables)
[x] I have added tests that prove my fix is effective or that my feature works
How to test this PR locally
Use following config
input {
stdin {}
}
output {
firehose {
# Use your prefered authentication method
region => "${YOUR_REGION}"
access_key_id => "${YOUR_ACCESS_KEY}"
secret_access_key => "${YOUR_SECRET_KEY}"
# Define a kinesis data firehose stream to test your events
delivery_stream_name => "test-stream"
}
}
Release notes
Add firehose output plugin
What does this PR do?
Send events to firehose with AWS SDK and the
put_record_batch
method.Events will be sent in batches, following the base
multi_receive_encoded
method.If number or size of events exceeds the AWS limits, sub-batches will be created.
The limits can be overriden by the users and they must define at least a
delivery_stream_name
configuration.Checklist
How to test this PR locally