search

RTICWDT / open-data-maker

make it easy to turn a lot of potentially large csv files into easily accessible open data
Other
64 stars 32 forks source link

Increase max page size of 100 to 1000 as there are 7112+ records currently #88

Closed zurbergram closed 4 years ago

zurbergram commented 4 years ago

Currently when performing a call to https://api.data.gov/ed/collegescorecard/v1/schools without selection parameters there is a total of 7112 records that can be retrieved however the max page size that is enforced is 100 which would require 72 rest calls to pull all of the data.

I propose increasing the max page size so larger sets of data can be pulled at once

The rest call (missing an api key) I am using is https://api.data.gov/ed/collegescorecard/v1/schools?api_key=&fields=id,ope8_id,ope6_id,school,latest&per_page=1000&page=71

JAFrederick commented 4 years ago

Thanks for the feedback and submitting a PR. We are going to keep the current MAX_PAGE_SIZE limitations for now. Keeping this API available and performant is absolutely vital. Increasing the MAX_PAGE_SIZE could introduce instability as returns are often very large.