Webhooks (e.g. a la github)

[rufuspollock]

Ability to add external services to notify on certain actions (e.g. dataset update, resource update, datastore change etc).

[rossjones]

I think @deniszgonjanin is looking at this over at https://github.com/deniszgonjanin/ckanext-webhooks

[rufuspollock]

That's just amazing - you think of something and already in progress!

Do we have user stories? Can I review or contribute in any useful way /cc @deniszgonjanin

[rufuspollock]

/cc @jqnatividad is this in the extensions registry?

[deniszgonjanin]

@rgrp yes, feedback is very welcome, especially it would be great if people weighed in on the design. The extension is really for two purposes:

• Allow users to subscribe to CKAN events without having to poll CKAN. Useful for all sorts of things - keeping your App data up-to-date or automatically refreshing visualizations you have made, etc... Polling is a pain since it requires cron or background tasks. Right not CKAN is pushing that complexity onto apps that integrate with the API. Instead, as an app developer, I would love to not have to do that for every app.
• Have a way to create fully decoupled service provider extensions.

The second item is in response to the datapusher, since there are a couple of fundamental problems with datapusher/ckan-service-provider:

• There is a CKAN component to the datapusher, and an external component, and they are tightly coupled to each other despite being two different codebases. This presents problems with versioning, maintenance.
• To create a different kind of service provider, you have to write a ckan extension that lives alongside your CKAN, as well as an outside service. Ideally you should only have to write one.
• CKAN communicates with datapusher within the request/response lifecycle, which blocks CKAN if datapusher is busy/non-responsive/down.

ckanext-webhooks is a proposed solution, in that it works similar to other webhook services, such as Github's. It's up to the service to register with CKAN on the events it would like to receive, and CKAN simply fires off a webhook when that event happens. This has some nice properties:

• You don't need to write a separate ckanext for your service, decoupling the two code-bases.
• This also means any user can hook into CKAN events, not only those with ability to install extensions.

To deal with the blocking problem, the extension uses a tornado event loop. CKAN notifies tornado of webhooks, and continues doing other things while tornado notifies all the subscribers. This is necessary, as there is no good facility in CKAN to fire off asynchronous events or perform background tasks. I considered integrating celery, but the tornado solution is much simpler, has minimal dependencies and works well.

There are still certain things missing - there is a todo at the bottom of the repo. I am using the extension and it's been working quite well despite some of the missing pieces.

My long term hope is to refactor datapusher to use something like this for 2.4, which would make the CKAN core a bit cleaner and also allow for more services to be built on top. One downside is pointed out by @rossjones here . Since CKAN would be decoupled from services using webhooks, you would not be able to see the datapusher job status page on every resource page. But there are some possible solutions to that, too.

[rossjones]

I stumbled across https://github.com/gophergala/rehook recently - it looks very interesting as a separate component.

[jqnatividad]

Its now in extensions.ckan.org :)