MaxRetries Error while fetching 3.6 lakh records from Knack

[GuptaTanay]

Hi, I am trying to fetch around 3.6 lakh records from knack using knackpy.

I have used the to_csv function as below:-

app = knackpy.App(app_id=my_app, api_key=my_key) app.to_csv('object_19', out_dir = csv_dir)

app_id, my_key, csv_dir are referenced from the config file. I am getting the below errors:-

raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='thinkheyday-api.knack.com', port=443): Max retries exceeded with url: /v1/objects/object_19/records/?page=5&rows_per_page=1000 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000001525DC7E550>: Failed to establish a new connection: [Errno 11002] getaddrinfo failed'))

At first, I encountered this issue on page 26 out of 36, then it started coming on page 5.

Can anyone help me here? I have gone through the Knack docs that Knack has a limited the record extraction as a part of the API limit.

[GuptaTanay]

Now, it failed again on page 70.

[johnclary]

@GuptaTanay sorry to hear of you error. I have not tested this functionality with such a large number of records. This does look like you're exceeding the knackpy request limit, which is rare when downloading records. By chance are you running concurrent API requests against your app?

I wonder if you might have better luck trying to create multiple CSVs by using a filter parameter to query your records in smaller chunks. For example, if you have an ID sequence to fetch batches of 50k records. I hope that helps.

[GuptaTanay]

Hi @johnclary, thanks for replying. No, we are not making concurrent requests.

Yes, using filters could help, but I could see the filters only at one place in the documentation. But, sadly we don't have an ID column in the tables.

Does the below snapshot mean that rule 1 and rule 2 will be used in the OR condition?

Would you be able to share a detailed link to filters? I will try to write a filter and get back to you for errors.

Thanks in advance!

[johnclary]

@GuptaTanay

Does the below snapshot mean that rule 1 and rule 2 will be used in the OR condition?

The example above will return records matching either condition. Here are the Knack docs on filtering.

I'm surprised you're hitting a rate limit on you downloads. Are you sure you're not exceeding your daily request quota? 360k records would amount to 360 requests.