search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.51k stars 1.53k forks source link

Add support for auto-concurrency to `aws_sqs` source #13322

Open bruceg opened 2 years ago

bruceg commented 2 years ago

The aws_sqs source has support for client concurrency, running more than one polling client thread at once. This defaults to one client per CPU currently. However, those threads will frequently be idle, resulting in excess unnecessary polling, and also cannot be scaled beyond the configured concurrency. This source could adapt the auto-concurrency idea from gcp_pubsub (#13240) and scale its connections dynamically as there is discovered need for higher loading. This would allow us to scale our use of the SQS API to current conditions rather than requiring configuration for the worst case loading.

nhlushak commented 11 months ago

Such option would be a very nice feature, since SQS is payed-per-request service and queue ingesting could differ depending on overall system load.