search

bookwhen / opendata

Documentation, Changelog and Issues related to the Bookwhen RPDE endpoint
0 stars 1 forks source link

Higher page size? #10

Closed lukehesluke closed 4 years ago

lukehesluke commented 4 years ago

Hi,

Love the feed. Love the docs. I just had one suggestion: Would it be possible to increase the page size?

It's currently set to 10, which means that it takes a very long while for crawlers (e.g. crawlers which use a cronjob) to get to the end of it. This means that Bookwhen data in these aggregators will be less up to date.

If it was set to 500, that would be ideal. This is also what RPDE recommends (https://www.w3.org/2017/08/realtime-paged-data-exchange/#modified-timestamp-and-id).

You could implement this as a limit query parameter, which is an optional field in RPDE. e.g. &limit=500

Thanks :blush:

Luke

jopotts commented 4 years ago

Hi. I've increased the page size and removed old data, so the import should now be a lot quicker. The page size isn't 500 as you suggested, as that would result in long response times, smaller response times are better for the servers, and should ultimately result in a similar total time. I think it's now at 50 so give that a go.

domfennell commented 4 years ago

Happy New Year, @jopotts. Thanks for sorting this. It's made a big difference to the speed at which our platform can ingest your data. Much appreciated.