search

auth0 / jupiterone-python-sdk

A Python client for jupiterone.io
MIT License
10 stars 17 forks source link

Allow Query with Limit > 250 #9

Open ruidan-li opened 3 years ago

ruidan-li commented 3 years ago

Description

With the default rate limit (30 per minute) and the default upper bound of query limit (250), the maximum number of results of a query is strictly bounded (250 * 30 = 7,500 per minute). So when the expected number of results is larger than this upper bound, then there is almost no way to get the full results (unless the response is slow and thus it doesn't hit the API 30 times per minute).

Even though the limit in this SDK is not checked, the 250 upper bound is enforced at JupiterOne side:

JupiterOneApiError: [{'code': 'INTERNAL_SERVER_ERROR', 'message': 'Error executing J1QL query: "limit" must be a value between 1 and 250 (inclusive). Received: 500 (code=SYNTAX_ERROR)', 'locations': [{'line': 2, 'column': 3}], 'path': ['queryV1']}]

Reproduction

In the current design, when the expected result is larger than 7,500, and the response time is fast enough to allow >= 30 query execution, a JupiterOneApiRetryError is guaranteed to be triggered.

Environment

ruidan-li commented 3 years ago

It would be very helpful if there could be another variable specifying a start_skip value, so that when rate limit is reached and the exception is raised, I can wait for a certain time and query again with a none zero skip.

bsmith-auth0 commented 3 months ago

Hi @ruidan-li ,

We will be archiving this repo in favor of the new official JupiterOne Python SDK. If this issue still remains, please open an issue on the new repo.