logstash-plugins / logstash-integration-aws

Apache License 2.0
7 stars 17 forks source link

Add Firehose output #47

Open pierredeman opened 3 months ago

pierredeman commented 3 months ago

Release notes

Add firehose output plugin

What does this PR do?

Send events to firehose with AWS SDK and the put_record_batch method.
Events will be sent in batches, following the base multi_receive_encoded method.
If number or size of events exceeds the AWS limits, sub-batches will be created.
The limits can be overriden by the users and they must define at least a delivery_stream_name configuration.

Checklist

How to test this PR locally

  1. Use following config
    input {
    stdin {}
    }
    output {
    firehose {
    # Use your prefered authentication method
    region => "${YOUR_REGION}"
    access_key_id => "${YOUR_ACCESS_KEY}"
    secret_access_key => "${YOUR_SECRET_KEY}"
    # Define a kinesis data firehose stream to test your events
    delivery_stream_name => "test-stream"
    }
    }
  2. Send some input data
  3. Verify all is received by firehose
cla-checker-service[bot] commented 3 months ago

💚 CLA has been signed