Closed sumo-drosiek closed 2 years ago
Instead of doing this, couldn't we just make the requeue non-blocking and drop data if there's no space left in the queue? Do we know what the right thing to do in general is for a Logstash output if it can't send data fast enough? Should it block, return an error, or something else?
AFAIK if queue is full it is going to backpressure logstash, so no new events are going to be put into queue until it has space
Should it block, return an error, or something else? not sure, so leaving main architecture as it is
I want to avoid changing architecture or behavior if there is no need to do that
I'm merging this PR as we need bugfix urgently. We can continue discussion here and add changes as separate PRs
In order to send data simultaneously, one of two approaches can be used here:
This PR keeps the async approach and prevent deadlock by adding additional queue (sized twice as worker numbers)
I wanted to add minimal set of changes in order to fix #71