Is RPDE feed valid? tool

[MelanieAbraham]

Validation of whether an RPDE feed is valid. I think this is a separate tool (or page on the existing tool) though, as it needs to operate differently.

Todo:

• [x] Worker / RabbitMQ research
• [x] UX prototype
• [x] Extra rules as below
  • [x] Check for required properties
  • [x] Prompt for item data depending on type
  • [x] Warning if modified / timestamp are not integers
  • [x] Error if afterId is not present when afterTimestamp is
• [x] Tests
  • [x] HttpContentTypeRule
  • [x] HttpStatusRule
  • [x] LastPageUrlIsSelfRule
  • [x] NoLastPageItemsRule
  • [x] AfterChangeNumberRule
  • [x] AfterTimestampRule
  • [x] DeletedItemsRule
  • [x] DuplicateItemsRule
  • [x] MinimumItemsRule
  • [x] NextPageValidRule
  • [x] ValidJsonObjectRule
  • [x] LastPageCheckRule

[nickevansuk]

As discussed with @ldodds, suggested checks as follows:

Crawl the first 20 pages:

• Does the next url work as expected and return a valid page - it should never result in a 500 error
• If afterTimestamp is used, and "modified" is an integer, afterTimestamp must also be an integer
• If afterChangeNumber is used it must be an integer and "modified" must be an integer
• Does the afterTimestamp or afterChangeNumber of the next url always increase with each new page - if not the query has likely been badly implemented. Most feeds should use integer for this, so you can check they increase numerically afterTimestamp (for integer "modified") or lexicographically (for string "modified").
• There should be "deleted" items in the feed. If these are missing, it is likely they have not been implemented correctly. Display a warning: "No deleted items detected. Please check that deleted items have been implemented correctly."
• The next URL should be an absolute not relative URL
• Are all responses returned with header Content-Type: application/json
• Items should not appear more than once in the first 20 pages - build up a hashmap of ids and error if duplicates are detected
• Pages should contain at least 500 items (this is a warning rather than an error)

Last Page check:

• afterTimestamp: If "modified" is an integer, put a high integer in for afterTimestamp to return the last page. If "modified" is not an integer return a warning saying Last page check could not be completed, as "modified" and "afterTimestamp" values should be integers. It is recommend that these are integers to be compatible with future version of RPDE. (N.B. the spec currently allows for strings to be used for "modified", but a future spec will likely make integers a must here).
• afterChangeNumber: Put a high integer in for afterChangeNumber to return last page
• Check that the next url of last page points to itself
• Check that the items array of the last page is empty

[nickevansuk]

Also further feedback based on demo this morning... for the modelling spec checks within the RPDE feed, either a) check all items in the feed or b) check the first x but skip deleted items and display a warning saying no items found to validate if the first page contains only deleted items, as otherwise a feed's first page will very likely not be able to be validated for live feeds with 10 deleted items

[petewalker]

• Licence property
• Next page property
• Items property
  • Should be non-empty if not on last page

[ldodds]

See also 4.4.2 https://www.w3.org/2017/08/realtime-paged-data-exchange/#-item-

[petewalker]

Prompt for item data depending on type (point to model validator if has oa context, point to modelling spec if not)

[nickevansuk]

Quick update to these rules:

• The afterTimestamp of the next url should always increase with each new page, UNLESS it is the same as the previous page, in which case the afterId should increase (either numerically if it can be parsed to an int, or lexicographically if not)