Cooperative clients trip rate limit causing false alarm positives

DataBiosphere / azul

Metadata indexer and query service used for AnVIL, HCA, LungMAP, and CGP

Apache License 2.0

6 stars 2 forks source link

Cooperative clients trip rate limit causing false alarm positives #6001

Open dsotirho-ucsc opened 6 months ago

dsotirho-ucsc commented 6 months ago

My demo of DataBiosphere/azul-private#125 caused a azul-waf_rate_blocked-dev alarm. Since this scenario represents a cooperative use of the API (the rate limit is only briefly exceeded and the client obeys the retry-after header), we should ideally not get an alarm.

dsotirho-ucsc commented 6 months ago

Assignee to consider next steps.

hannes-ucsc commented 5 months ago

Adjust alarm threshold to allow for n clients exhibiting this behavior without triggering an alarm. Make n configurable, defaulting to 1.

For demo, repeat demo for https://github.com/DataBiosphere/azul-private/issues/125 and show that no alarm is triggered.

To decide priority I would like to sit on this for a while.