search

meraki / dashboard-api-python

Official Dashboard API library (SDK) for Python
MIT License
289 stars 151 forks source link

502 Bad Gateway for getNetworkClientsUsageHistories with large page #190

Closed j-adamczyk closed 1 year ago

j-adamczyk commented 2 years ago

I am getting the following error when using Python SDK (synchronous version) with getNetworkClientsUsageHistories endpoint with large number of entries per page, i.e. above 64 (docs say that 1000 is the limit):

  File "/home/airflow/.local/lib/python3.8/site-packages/meraki/api/networks.py", line 402, in getNetworkClientsUsageHistories
    return self._session.get_pages(metadata, resource, params, total_pages, direction)
  File "/home/airflow/.local/lib/python3.8/site-packages/meraki/rest_session.py", line 378, in _get_pages_legacy
    response = self.request(metadata, 'GET', url, params=params)
  File "/home/airflow/.local/lib/python3.8/site-packages/meraki/rest_session.py", line 243, in request
    raise APIError(metadata, response)
meraki.exceptions.APIError: networks, getNetworkClientsUsageHistories - 502 Bad Gateway, <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></cente

For very small pages (e.g. 3, 16, 32) it works, but returns only the first page, even when using total_pages="all". It does work in all cases (<=32 and >32 client IDs) from Postman.

Whan can I do about this?

j-adamczyk commented 2 years ago

I checked that Postman uses n244 shard, instead of API Gateway. SDK does not work probably because routing is bugged.

Postman base URL that works: https://n244.meraki.com/api/v1. SDK base URL that does not work: https://api.meraki.com/api/v1.

After setting self.base_url = "https://n244.meraki.com/api/v1" in RestSession class instance it works, but only in some cases.

EDIT: it also requires asking for at most 500 client IDs at one time to work. This is not documented at all.

TKIPisalegacycipher commented 2 years ago

Thanks for reporting this @j-adamczyk. In Postman, if you change n244 to api in your base URL, does Postman have the same problem?

j-adamczyk commented 2 years ago

@TKIPisalegacycipher yes, Postman does work after that change, but SDK still does not.

TKIPisalegacycipher commented 2 years ago

Thanks @j-adamczyk. Since the issue doesn't happen with api.meraki.com via Postman, that tells me it's not an api.meraki.com-specific issue.

I checked out the API endpoint itself (i.e. the docs link you shared) and by design, it returns an array, wherein each item also contains a nested array called usageHistory. The perPage attribute will only limit the size of the top-level array. The timespan query param, however, will limit the size of usageHistory. I wonder if this issue is related to the Python requests library handling a certain payload size in conjunction with an auth redirect when the usageHistory array is very long, e.g. when the timespan is large.

When you see this issue, what is the timespan you are using?

While using api.meraki.com, are you able to reproduce this problem with short timespans (e.g. 60 for 60 seconds)?

j-adamczyk commented 2 years ago

I am using 24 hours timespan. Interestingly, with 24 hours timespan and page size 500 (so quite large payload) it works with n244, but not with api.

For 60 second timespan and page size 500, I get the same error, and even for timespan 30. However, combination of page size 32 and timespan of 60 seconds returns all pages.

TKIPisalegacycipher commented 1 year ago

Would the request URI length happen to be over 8 KB?

TKIPisalegacycipher commented 1 year ago

My best guess at this time is that there's an 8KB URI size limit, but this is happening at the API level, not the SDK level. This is not something we can fix from the SDK side, so closing this out.

I would recommend splitting those requests into separate queries where URIs are <8KB.