search

google / slo-generator

SLO Generator computes SLIs, SLOs, Error Budgets and Burn Rates from supported backends, then exports an SLO report to supported targets.
Apache License 2.0
475 stars 75 forks source link

[Datadog] API limits reached #243

Open bkamin29 opened 1 year ago

bkamin29 commented 1 year ago

Hello,

I'm facing a Datadog API rate limit issue when using the SLO Generator. This problem seem due to a rate-limit implemented on Datadog APIs : https://docs.datadoghq.com/api/latest/rate-limits/

I don't know if from a SLO Generator point of view, we can be able to reduce our amount of API call to compute SLOs.

lvaylet commented 1 year ago

Hi @bkamin29,

Have you discussed Adeo' s architecture with @clement-pruvot?

Off the top of my head, they managed to generate an artifical rate limiting on the number of requests to the Datadoc API by calling the containerized SLO Generator (in Cloud Run) from an intermediate Cloud Function.

ocervell commented 1 year ago

Yes, that's what they did. We might help though by adding retries within the slo-generator, like we did for Dynatrace here.