RPDE Orders Feed ordering error

[lukehesluke]

(This error was reported by Broker Microservice)

FATAL ERROR: RPDE Validation Error(s) found on RPDE feed OrdersFeed (auth:secondary) page "https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1729596298655551&afterId=6c81ae8e-4272-48c9-821d-56211754bba9":

• When using the Modified Timestamp and ID Ordering Strategy, the combination of afterTimestamp and afterId in the next url must always increase with each new page, as the primary query sorts first by modified and then by id. The next URL of this page is "https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1729596298650499&afterId=6c81ae8e-4272-48c9-821d-56211754bba9"

Broker was harvesting a page with URL https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1729596298655551&afterId=6c81ae8e-4272-48c9-821d-56211754bba9, and it contained a next URL which pointed to https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1729596298650499&afterId=6c81ae8e-4272-48c9-821d-56211754bba9. This latter URL has a lower timestamp than the former URL. This should be impossible as items should be ordered by timestamp in ascending order.

RPDE ordering not working as expected could lead to all sorts of issues, like data being missed by the client, the client being sent into a loop and so never getting new data, getting data in the wrong order, etc.

It also stops Test Suite from passing as soon as it's spotted

[nathansalter]

Unsure why this would be happening, did it create a loop of the pages or was it just a one-off? Can you replicate this?

[nickevansuk]

Hey @nathansalter - having tested this too I can confirm this issue occurs consistently (at least 10 runs in a row).

I used the following zsh script to run the test suite multiple times:

until npm start; do
  echo "Retrying npm start..."
done

Each run the test suite failed with a version of the above, such as:

FATAL ERROR: RPDE Validation Error(s) found on RPDE feed OrdersFeed (auth:secondary) page "https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1731081413402516&afterId=a441912c-dd48-434f-8a06-3983a933d7b3":
- When using the [Modified Timestamp and ID Ordering Strategy](https://www.openactive.io/realtime-paged-data-exchange/#modified-timestamp-and-id), the combination of `afterTimestamp` and `afterId` in the next url must always increase with each new page, as the primary query sorts first by `modified` and then by `id`. The next URL of this page is "https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1731081413402516&afterId=a441912c-dd48-434f-8a06-3983a933d7b3"
- The `next` property `"https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1731081413402516&afterId=a441912c-dd48-434f-8a06-3983a933d7b3"` must not be equal to the current page URL `"https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1731081413402516&afterId=a441912c-dd48-434f-8a06-3983a933d7b3"` where there are more than zero `items` in the page. The last item in each page must be used to generate the `next` URL.

FATAL ERROR: RPDE Validation Error(s) found on RPDE feed OrdersFeed (auth:primary) page "https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1731083652551647&afterId=16496bd0-a2bb-4d83-b0c1-caaa06af38a7":
- When using the [Modified Timestamp and ID Ordering Strategy](https://www.openactive.io/realtime-paged-data-exchange/#modified-timestamp-and-id), the combination of `afterTimestamp` and `afterId` in the next url must always increase with each new page, as the primary query sorts first by `modified` and then by `id`. The next URL of this page is "https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1731083652551647&afterId=16496bd0-a2bb-4d83-b0c1-caaa06af38a7"
- The `next` property `"https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1731083652551647&afterId=16496bd0-a2bb-4d83-b0c1-caaa06af38a7"` must not be equal to the current page URL `"https://imin.strmln.uk/api/open-active/orders-rpde?afterTimestamp=1731083652551647&afterId=16496bd0-a2bb-4d83-b0c1-caaa06af38a7"` where there are more than zero `items` in the page. The last item in each page must be used to generate the `next` URL.