Bring RPDE in line with JSON-LD and remove duplication

Proposer

ODI

Requirements

Bring RPDE in line with JSON-LD
Remove duplication of properties
Allow feeds to be combined with Opportunity API (https://github.com/openactive/opportunity-api/issues/10)

Proposal

Replace "kind"with JSON-LD "@type"
Use JSON-LD "@id"
Augment JSON-LD with "@modified" to allow any entity to have a modified date
Augment JSON-LD with "@deleted" to allow any entity to have a "deleted" state (using a boolean instead of the current updated|deleted field as there are only two possible values for that field).
Use @ for type, id, modified and deleted, to distinguish between control properties and data properties.
Items with "@deleted": true MUST only include "@id", "@modified" and "@deleted" properties (note that the Opportunity API could also return such small response for objects that have been deleted, for consistency).
"@id" and "@modified" are also used in the Opportunity API to allow items retrieved to be comparable to the those in the feed.
Make "items" singular in line with schema.org naming contentions
Add "type": "RealtimePage" and "@context": "https://openactive.io/" to the base of the feed
"@modified" MUST use ISO 8601 format with sub-millisecond precision

Example

{
  "@context": "https://openactive.io/",
  "type": "RealtimePage"
  "next": "http://api.letsride.co.uk/public/v1/rides?afterModified=2018-09-30T18%3A45%3A10.263Z&afterId=http%3A%2F%2Fwww.example.com%2Fevents%2F2710",
  "item": [
    {
      "@id": "http://www.example.com/events/2710",
      "@type": "SessionSeries",
      "@modified": "2018-09-30T18:45:10.242Z",
      "name": "Speedball",
      "offers": [
        {
          "@type": "Offer",
          "@id": "https://example.com/events/2710#/offers/878",
          "validThrough": "2018-10-29T11:00:00Z",
          "validFrom": "2018-10-01T11:00:00Z",
          "description": "Winger space for Speedball.",
          "name": "Speedball winger position",
          "price": 10.00,
          "priceCurrency": "GBP",
          "isCancellable": true,
          "cancellationValidUntil": "2018-10-28T11:00:00Z"
        }
      ],
      "duration": "PT1H",
      "organizer": {
        "@type": "Organization",
        "name": "Central Speedball Association",
        "url": "http://www.speedball-world.com",
        "termsOfService": { "type": "TermsOfService", "url": "url"}
      },
    },
    {
      "@id": "http://www.example.com/events/2710",
      "@type": "SessionSeries",
      "@modified": "2018-09-30T18:45:10.263Z",
      "@deleted": true
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}

Open questions

"modified" as ISO 8601 format with sub-millisecond precision - is this necessary? As long as the format is consistent between the Opportunity API and RPDE feed could we still use native integer changeNumber as the timestamp instead?

Just to note that schema.org has since introduced https://schema.org/DataFeed and https://schema.org/DataFeedItem, which are both applicable here.

A suggested mapping of the current RPDE terms to schema.org (also picking up on the points in https://github.com/openactive/realtime-paged-data-exchange/issues/54), is as follows:

RPDE 1.0	Class	Property	Type
`next`	`oa:RealtimePagedDataFeed`	`oa:nextUrl`	`schema:URL`
`items`	`oa:RealtimePagedDataFeed`	`schema:dataFeedElement`	Array of `schema:DataFeedItem`
`license`	`oa:RealtimePagedDataFeed`	`schema:license`	`schema:URL`
	`oa:RealtimePagedDataFeed`	`schema:schemaVersion`	`schema:URL`
`state`	`schema:DataFeedItem`	`oa:itemDeleted`	`schema:Boolean`
`kind`	`schema:DataFeedItem`	`schema:item.@type`	`schema:Property`
`id`	`schema:DataFeedItem`	`schema:identifier`	`schema:Text` OR `schema:Integer`
`modified`	`schema:DataFeedItem`	`schema:version`	`schema:Integer`
`data`	`schema:DataFeedItem`	`schema:item`	`schema:Thing`

A few notes on this mapping:

The purpose of this mapping is to bring RPDE in line with schema.org and JSON-LD, without making any functional changes to the now thoroughly tried-and-tested RPDE mechanics.
oa:RealtimePagedDataFeed subclasses schema:DataFeed to indicate the specific type of data feed this represents, which aids validation and tooling.
state is replaced by a boolean oa:itemDeleted, as RPDE only specifies two states - so this is a more compact representation. Note that oa:itemDeleted defaults to false, so is not required to be present in the scenario that was previously described as state of updated.
identifier would still be used within RealtimePagedDataFeedItem rather than using item.@id in order to maintain the ?afterId= parameter behaviour without require full URL encoding (and also in line with https://github.com/openactive/realtime-paged-data-exchange/issues/87). ?afterId= should also be renamed to ?afterIdentifier=.
schemaVersion could additionally be added to reference the model in use by the feed (building on suggestions in https://github.com/openactive/realtime-paged-data-exchange/issues/86).
The use of RPDE should be specified as a mime type via the Content-Type header, in line with the tooling - although this feed would be fully compatible with application/ld+json and the root @type of RealtimePagedDataFeed infers RPDE, the underlying transport mechanism of RPDE should still be specified via Content-Type ala application/atom+xml.
As per https://github.com/openactive/realtime-paged-data-exchange/issues/89 ordering remains an integer to allow it to be compatible with both afterId and afterChangeNumber ordering strategies.
For items, version is preferred to modified as it does not imply any relationship to time, and respects the invariant (https://github.com/openactive/realtime-paged-data-exchange/issues/80#issuecomment-367261937) - this maintains the simplicity of implementation as data consumers can treat both ordering strategies in the exactly the same way. ?afterChangeNumber= and ?afterTimestamp= parameters should not be renamed to ?afterVersion=, as their separation is useful for validation and debugging.
kind is replaced by item.@type, as there is no need to duplicate the property within DataFeedItem when it is mandatory within Thing contained in the item. This means that deleted items require an empty Thing with only @type specified, to preserve the ability to combine multiple kinds within one feed, and for the data consumer to easily store these in separate places.
Suggest we remove the dependency on the Opportunity API for this next version, partly as the Opportunity API is not on the near-term roadmap, and partly as the idea of combining real-time feed data with the latest search results from an Opportunity API is an architectural nightmare. Better for implementers to focus on improving the performance of their RPDE implementation.
We should include cache headers in this next version of the RPDE spec, including the use of the expires and Cache-Control headers to hint at data consumer polling frequency and inform the CDN, as per https://github.com/openactive/realtime-paged-data-exchange/issues/79. Such practices are already in use in production, so this is just formalising this into the spec.

{
  "@context": "https://openactive.io/",
  "@type": "RealtimePagedDataFeed",
  "nextUrl": "http://www.example.com/api/rpde/sessions?afterTimestamp=1453931925&afterIdentifier=%7Bd97f73fb-4718-48ee-a6a9-9c7d717ebd85%7D",
  "dataFeedElement": [
    {
      "@type": "DataFeedItem",
      "identifier": "{c15814e5-8931-470c-8a16-ef45afedaece}",
      "version": 1453931101,
      "item": {
        "@type": "Event",
        "@id": "https://bookingsystem.com/hulahoop/e/ev-ssyp-20160509191500",
        "identifier": "session-01jz93i3k1p3",
        "startDate": "2016-05-09T18:15:00Z",
        "endDate": "2016-05-09T19:15:00Z",
        "remainingAttendeeCapacity": 0,
        "url": "https://bookingsystem.com/hulahoop/e/ev-ssyp-20160509191500?r=oa",
        "attendanceCount": 29,
        "location": {
          "@type": "Place",
          "identifier": "location-jbnentuhxsnp",
          "url": "https://leisurecentre.org.uk/leisure-centre/camden/kentish-town-sports-centre",
          "name": "Kentish Town Sports Centre",
          "address": {
            "@type": "PostalAddress",
            "addressLocality": "Camden",
            "addressRegion": "London West",
            "postalCode": "NW5 3DU",
            "streetAddress": "Kentish Town Sports Centre"
          }
        }
      }
    },
    {
      "@type": "DataFeedItem",
      "identifier": "{c15814e5-8931-470c-8a16-ef45afedaece}",
      "version": 1453931925,
      "itemDeleted": true,
      "item": {
        "@type": "Event"
      }
    }
  ],
  "schemaVersion": "https://www.openactive.io/modelling-opportunity-data/2.0",
  "license": "https://creativecommons.org/licenses/by/4.0/"
}

openactive / realtime-paged-data-exchange

Bring RPDE in line with JSON-LD and remove duplication #90

Proposer

Requirements

Proposal

Example

Open questions