opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
262 stars 201 forks source link

New aggregate processor action - rate_limiter - to limit the number of events passed through aggregator in a given time period #2088

Closed kkondaka closed 1 year ago

kkondaka commented 1 year ago

Is your feature request related to a problem? Please describe. It would be nice to have rate limiting capability in aggregator actions to be able to limit the number of events while aggregating.

This is a sub-issue of issue #2015.

Describe the solution you'd like

Provide a new action rate_limiter that would limit the number of events per second as shown below

processor:
  aggregate:
    identification_keys:
      - # ... Identification keys for the metric ...
      action:
        rate_limiter: 
           events_per_second: 10
           drop_when_exceeds: true

The above config would limit the number of events per second to 10 and drops any excessive events that are rejected by the rate limiter. Setting drop_when_exceeds to false would result in rate limiter to block when an event could not be allowed to pass.

Additional context This action when combined with when condition of aggregation can be used to rate limit a events matching a specific condition. Rate limiting is one of the policies of tail sampling provided by OTEL - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md

oeyh commented 1 year ago

Resolved via #2090