PubSubHubbub

[alexdutton]

PSHH could provide a pleasant way to be informed of new data being ready to be ingested, and to notify downstream clients that data has changed.

OxTalks, and news feeds would be good candidates for using this stuff.

[martinfilliau]

+1

We had discussions about this with @davbo, we were envisioning m.ox subscribing to updates for OxPoints, as well as status.ox and talks.ox.

PubSubHubbub Core 0.4 supports arbitrary content but it's a working draft... I'd like to set up a pubsubhubbub hub to make some tests.

[alexdutton]

Before we can do PubSub, we need to be able to identify "interesting" resources, that people would be interested in subscribing to. These are some rough ideas for how to achieve all this.

We currently have humfrey.linkeddata.resource.Resource, which is used to override templates for particular rdf:types of resource, and to extend the query to the backend store to pull in related data where it's specifically about that resource (e.g. also return the address of a building when describing the building).

We can reuse the query extension bit to acquire a snippet of RDF. When this changes, we know to publish it to subscribers.

There will be two types of datasets:

• those updated as a single blob of RDF
• those with fine-grained, per-resource publishing.

These can be handled in the same way. Each time a graph is updated, we look for interesting resources in it ( { GRAPH ?g { ?x a ex:InterestingType } } UNION { ?g foaf:primaryTopic|foaf:topic ?x } }) and run the relevant queries against it. We compare the results to (a hash of?) what we had before, and if it's changed, we re-run the ElasticSearch indexer on it and re-publish it.

JSON can be provided from what gets indexed from ES, as could XML (transformed from JSON), and these could also be exposed at the doc pages. The Atom feeds would contain multiple content elements (XML, JSON, RDF/XML), and a summary element containing the rendered search result HTML.

Blank nodes make comparing RDF difficult (http://en.wikipedia.org/wiki/Graph_isomorphism_problem); suggestions welcome.

We should provide separate pubsub channels for:

• each type of interesting resource
• every interesting resource per dataset
• graph updates
• dataset updates

The PSHB hub should live at https://backstage.data.ox.ac.uk/stores/[store-name]/pubsubhubbub/

[martinfilliau]

In term of practical and immediate use cases, I’d like (as Mobile Oxford) to be able to subscribe to updates from OxPoints and courses data.

I’ve been looking at SparqlPush years ago (https://code.google.com/p/sparqlpush/) but your approach looks sensible; Never really used Fuseki but I remember ModelChangedListener in Jena (http://jena.apache.org/documentation/notes/event-handler-howto.html) might be interesting to look at?

Wondering if the PubSubHubbub Hub by itself should reside in data.ox depending on future usages? Depending on how goes events.ox, I’m sure Mobile Oxford would be happy to subscribe to its events, maybe status.ox could benefit of such a hub too...

[alexdutton]

XMPP might also work for real-time update subscriptions. Of course, it's another technology, not Web, and requires a daemon process…

[martinfilliau]

@alexsdutton yes, that's the place of projects such as https://github.com/tonyg/rabbithub from what I understand...

[alexdutton]

And another thing: http://code.google.com/p/sparqlpush/