search

turbot / steampipe-plugin-datadog

Use SQL to instantly query Datadog resources across accounts. Open source CLI. No DB required.
https://hub.steampipe.io/plugins/turbot/datadog
Apache License 2.0
12 stars 5 forks source link

Paginate queries to https://api.datadoghq.com/api/v1/monitor #47

Closed mdb closed 9 months ago

mdb commented 10 months ago

Because datadog_monitor does not perform paginated requests when calling the Datadog API /api/v1/monitor endpoint, the request can often fail for organizations with a large volume of monitors.

For example...

Using following query.sql:

SELECT
  name,
  'https://app.datadoghq.com/monitors/' || id AS URL
FROM
  datadog_monitor
WHERE
  query
LIKE
  '%ecs.fargate.cpu.limit%'

...results in an HTTP 504 from the Datadog API when the targeted organization has a large volume of monitors, which renders datadog_monitor unusable:

steampipe query query.sql

Error: datadog: 504 Gateway Time-out (SQLSTATE HV000)

+------+-----+
| name | url |
+------+-----+
+------+-----+

Describe the solution you'd like

I believe the HTTP 504 could be avoided if the steampipe-plugin-datadog issued the request(s) with a page_size query parameter, and fetched the monitors in smaller batches via distinct, per-page HTTP requests.

ParthaI commented 10 months ago

Thank you, @mdb, for highlighting this issue to us. Rest assured, we will investigate it thoroughly. 👍

mdb commented 9 months ago

Thank you, @mdb, for highlighting this issue to us. Rest assured, we will investigate it thoroughly. 👍

@ParthaI I hope you don't mind; I took a stab at a fix over in https://github.com/turbot/steampipe-plugin-datadog/pull/48.

misraved commented 9 months ago

Resolved in https://github.com/turbot/steampipe-plugin-datadog/pull/48