search

upshot-oa / opendata

Documentation, Changelog and Issues related to the Upshot RPDE endpoint
0 stars 0 forks source link

Higher page size? #2

Closed domfennell closed 3 years ago

domfennell commented 4 years ago

Hi,

We were just looking at your feed and had one suggestion: would it be possible to increase the page size?

It currently looks to be set to 10 items per page, which means that it can take a while for crawlers (e.g. crawlers that use a cronjob) to get to the end of it. For those crawling in this way, it could result in UpShot data being less up-to-date.

When we put your feed URL into the OpenActive RPDE Feed Validator, it provides the following warning: "Pages should contain at least 500 items to optimise the number of pages that a data consumer needs to download. Note this is only a recommendation, and actual default limit should be based on query performance."

You could implement this as a limit query parameter, which is an optional field in RPDE. e.g. &limit=500.

Many thanks,

Dom

zerolab commented 4 years ago

Hi Dom,

thank you for the suggestion. We do have a limit parameter that is currently capped at max 50. ie. if you query with &limit=500 it will still give you 50.

Will look at bumping up the upper limit after a discussion with the team.

domfennell commented 4 years ago

Great! Thanks, Dan. Look forward to hearing word after your team discussion.

Also, thanks for the info RE the limit. I tried adding &limit=50 to your RPDE URL and it still returned 10 items. Maybe I'm doing something wrong... @zerolab

zerolab commented 3 years ago

Changed the default to be 20 sessions, with a max of 100