snowplow / snowplow-elasticsearch-loader

Writes Snowplow enriched events from Kinesis to Elasticsearch
http://snowplowanalytics.com/
11 stars 18 forks source link

Use PutRecords to batch up failed inserts sent to kinesis #239

Closed istreeter closed 2 years ago

istreeter commented 2 years ago

Currently we use the singular PutRecord to send failed inserts one at a time to kinesis. It is much more efficient to use the plural PutRecords and send a batch in one go.

This change has become more important since addressing issue #238. Previously we used Futures when writing to kinesis, so we could tolerate inefficiencies because it did not block the main flow of execution. But since issue #238 we are going to start waiting for the result of writing to kinesis, instead of using Futures. Sending the batch in one go is the quickest way to unblock the main flow of execution.