SumoLogic / logstash-output-sumologic

Forward logstash data to Sumo Logic
Apache License 2.0
18 stars 15 forks source link

fix: remove deadlock possibility by adding resend_queue queue #73

Closed sumo-drosiek closed 2 years ago

sumo-drosiek commented 2 years ago

In order to send data simultaneously, one of two approaches can be used here:

This PR keeps the async approach and prevent deadlock by adding additional queue (sized twice as worker numbers)

I wanted to add minimal set of changes in order to fix #71

swiatekm commented 2 years ago

Instead of doing this, couldn't we just make the requeue non-blocking and drop data if there's no space left in the queue? Do we know what the right thing to do in general is for a Logstash output if it can't send data fast enough? Should it block, return an error, or something else?

sumo-drosiek commented 2 years ago

AFAIK if queue is full it is going to backpressure logstash, so no new events are going to be put into queue until it has space

Should it block, return an error, or something else? not sure, so leaving main architecture as it is

I want to avoid changing architecture or behavior if there is no need to do that

sumo-drosiek commented 2 years ago

I'm merging this PR as we need bugfix urgently. We can continue discussion here and add changes as separate PRs