search

DataDog / integrations-core

Core integrations of the Datadog Agent
BSD 3-Clause "New" or "Revised" License
908 stars 1.38k forks source link

Twistlock API calls do not consider pagination and are limited to the first 50 results #11149

Open kteague-tasktop opened 2 years ago

kteague-tasktop commented 2 years ago

The Twistlock integration does not handle paginated results from the Twistlock API. The Twistlock API is documented to only return the first 50 results from any given API call:

https://prisma.pan.dev/api/cloud/cwpp/21-08/api-limits/

We encountered this issue when using the datadog-agent (7.32.4, latest) to pull twistlock metrics and ingest into Datadog. We have an environment with 159 hosts and each host has 1 CVE, so expected to see a result of 159, but only saw 50 (we also have environments with 6 and 48 hosts and the results there were as expected).

We traced the call to get the /api/v1/hosts call made by the datadog agent to retrieving the JSON in the _retrieve_json method in the file twistlock/datadog_checks/twistlock/twistlock.py and the code doesn't deal with pagination, so is truncated to only the first 50 results.

fanny-jiang commented 2 years ago

Hi @kteague-tasktop, thanks for reporting this issue. I'm currently working on a fix for this.