Closed lukehesluke closed 4 years ago
Hi. I've increased the page size and removed old data, so the import should now be a lot quicker. The page size isn't 500 as you suggested, as that would result in long response times, smaller response times are better for the servers, and should ultimately result in a similar total time. I think it's now at 50 so give that a go.
Happy New Year, @jopotts. Thanks for sorting this. It's made a big difference to the speed at which our platform can ingest your data. Much appreciated.
Hi,
Love the feed. Love the docs. I just had one suggestion: Would it be possible to increase the page size?
It's currently set to 10, which means that it takes a very long while for crawlers (e.g. crawlers which use a cronjob) to get to the end of it. This means that Bookwhen data in these aggregators will be less up to date.
If it was set to 500, that would be ideal. This is also what RPDE recommends (https://www.w3.org/2017/08/realtime-paged-data-exchange/#modified-timestamp-and-id).
You could implement this as a
limit
query parameter, which is an optional field in RPDE. e.g.&limit=500
Thanks :blush:
Luke