cisco_duo: Make retry options configurable in CEL-based datastreams

kcreddy commented 9 hours ago

Proposed commit message

Cisco Duo Admin API has very low API request limit (Ex: not more than 1 request per minute for Authentication logs). Reaching this limit would lead to 42901 from the API. The response also doesn't include a Retry-After header, which leads to a default back-off retry attempts. This subsequently leads to further 42901 as the limit is already reached.

This PR allows users to set optional retry configuration allowed by CEL input for CEL-based data-streams. When number of retries are reduces, this helps in reducing the number retry requests being made when already reached 429.

Reference: https://help.duo.com/s/article/1338?language=en_US https://duo.com/docs/adminapi#authentication-logs https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_retry_max_attempts

[!NOTE] This only reduces 42901 Too Many Requests from the retry attempts. The users have to tune Interval and Rate limit to avoid the 42901 from the initial request (non-retry).

Checklist

[ ] I have reviewed tips for building integrations and this pull request is aligned with them.
[ ] I have verified that all data streams collect metrics or logs.
[ ] I have added an entry to my package's changelog.yml file.
[x] I have verified that Kibana version constraints are current according to guidelines.
[ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices

Screenshots

elastic-vault-github-plugin-prod[bot] commented 8 hours ago

:rocket: Benchmarks report

Package `cisco_duo` :+1:(3) :green_heart:(2) :broken_heart:(3)

Expand to view

Data stream | Previous EPS | New EPS | Diff (%) | Result ----------- | ------------ | ------- | -------- | ------ `activity` | 3649.64 | 2857.14 | -792.5 (-21.71%) | :broken_heart: `telephony_v2` | 22222.22 | 14925.37 | -7296.85 (-32.84%) | :broken_heart: `trust_monitor` | 16393.44 | 9345.79 | -7047.65 (-42.99%) | :broken_heart:

To see the full report comment with /test benchmark fullreport