snowplow / snowplow-elasticsearch-loader

Writes Snowplow enriched events from Kinesis to Elasticsearch
http://snowplowanalytics.com/
11 stars 18 forks source link

NSQ executor should periodically flush buffer #256

Closed istreeter closed 1 year ago

istreeter commented 1 year ago

Issue raised by snowplow/snowplow-mini#378. This issue relates only to the nsq source.

To improve Mini performance we want to use larger batches when sending events to Elasticsearch. But with low event volumes, we don't want events to get stuck in the buffer, while waiting for the buffer to fill.

The solution is to have an asynchronous process that periodically flushes the buffer.