Feature/improvement suggestions for ckanext-realtime

[reederz]

I have been working on ckanext-realtime (https://github.com/alexandrainst/ckanext-realtime) for some time now and I have reached the initial goal for it. If you enable this extension on your CKAN instance, client applications can monitor changes to datastores in realtime. In the picture below you can see how the realtime datastores are enabled with this plugin.

1. In order to emit realtime events, each "realtime datastore" table has a PostgreSQL trigger which sends NOTIFY events to datastore_listener script. The insertion of the trigger to datastore tables happens automatically: first time someone is interested to monitor a particular datastore, that datastores gets the trigger.
2. datastore_listener script sends datastore events (insert/update/delete) to ckan through Action API function _realtime_broadcastevent.
3. CKAN and WebSocket server(s) are integrated using Redis Pub/Sub messaging where CKAN is the publisher and WebSocket server(s) are the subscribers. This way, every event sent to _realtime_broadcastevent API function ends up in WebSocket servers.
4. Clients can subscribe to realtime events of certain datastores (pub/sub) using CkanRT.js library (included in ckanext-realtime repo).

So, right now it is possible for client apps to observe insert/delete/update events of certain datastores. You can find a demo app here http://gatesense.com/realtime/examples/ex2/ .

Other things that could be implemented:

• dataset level realtime events (new resources in datasets, dataset metadata change, etc.)
• direct integration into CKAN UI
• access control
• anything else?

The project was initiated by Alexandra Institute (http://www.alexandra.dk/uk/pages/default.aspx) and later adopted by Gatesense community (http://gatesense.com/) where Alexandra is one of the contributors. It is an effort to research realtime data publishing in the context of IoT and eventually gear CKAN towards being "IoT enabled" (though this may require additional extensions). That said, this extension has not been used on "real world applications" yet and it may not cover your specific use case. Therefore, we welcome community's collaborations and suggestions on how to improve it.

[stevenlivz]

This is excellent :-)

[rufuspollock]

+1 very cool!

[rufuspollock]

@reederz this is great. Do you think you could provide a short 1-2 paragraphs summarizing this work suitable for a broad audience (so not too technical) for inclusion in the next newsletter - you can put your info in the newsletter scratchpad google doc

[reederz]

@rgrp yes, I can do that. I reckon that I should do that today because you are releasing the newsletter soon, right?

[rufuspollock]

@reederz yes - i you could add something today or tomorrow that would be great.