I think we may be getting some failed requests as PlanIt now has an adjusted set of limits, to both the requests per minute, the total MB per hour, and the size of the response which is now 1MB. I have verified that a page_sz request of 500 sometimes fails due to being just over the 1MB limit.
The page size limit should be changed to 250 which I confirm is reliable.
Note that the total available applications that comes back in the response has now been confirmed as an estimate and not an exact number. This may be being improved as I believe there is a truncation bug in the API's pagination system which I have reported upstream.
mvl22
commented
3 months ago
I think we may be getting some failed requests as PlanIt now has an adjusted set of limits, to both the requests per minute, the total MB per hour, and the size of the response which is now 1MB. I have verified that a page_sz request of 500 sometimes fails due to being just over the 1MB limit.
The size and wait period are defined in three places in: https://github.com/cyclestreets/cyclescape/blob/master/lib/planning_application_worker.rb and in the specs: https://github.com/cyclestreets/cyclescape/blob/master/spec/lib/planning_application_worker_spec.rb
The page size limit should be changed to 250 which I confirm is reliable.
Note that the total available applications that comes back in the response has now been confirmed as an estimate and not an exact number. This may be being improved as I believe there is a truncation bug in the API's pagination system which I have reported upstream.