search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.57k stars 1.54k forks source link

Propagate backpressure as 429s for HTTP-based sources #7186

Open jszwedko opened 3 years ago

jszwedko commented 3 years ago

A user was curious about how backpressure propagates to the splunk_hec source if the downstream can't keep up. My understanding of the backpressure, is that it'll manifest as the splunk_hec source taking longer to process requests as it blocks trying to push events onto the output channel.

This can have the undesirable effect of vector consuming more resources holding onto all of these connections. I think it'd be useful for Vector to be able to start returning a 429 at a certain (maybe configurable) point to shed load without taking up significant resources. Maybe this would just be a max_connections parameter on HTTP sources.

jdrouet commented 3 years ago

@binarylogic what does the label "needs approval" mean here?

binarylogic commented 3 years ago

Ah, sorry, removing. It typically means that the issue has open questions and needs approval from others on the team before work can begin. It's a way to avoid unnecessary rework. But I don't think we want to do this right now. The multi-pipeline change is priority.