Webhooks

[JMLX42]

3rd party Apps - especially if they provide there own user database - are more than likely to need to know what voter participated to what vote. For example, LaPrimaire.org must be able to send reminders to the people who did not complete the voting process.

One could use blockchain events or we could provide a way for our smart contract to call a specified sub-contract. But the blockchain stores only the ballot values and has no information about the "real" user. This method cannot apply here but could be used for monitoring the voting process itself instead of the state of the users.

Thus, 3rd party Apps integrating Cocorico should be made aware of vote/ballot events using webhooks. When registering an App, the developer should be able to provide web hooks for the following events (at least):

• [x] a ballot has been successfully recorded on the blockchain
• [x] an error happened while trying to record a ballot
• [x] automatically retry failed (non 2XX) webhook calls
• [ ] send an e-mail when a webhook call failed for more than 1 day

Those webhooks must guarantee the privacy of the user and avoid sending private data. For example, the webhook can tell you that the user has successfully voted, but will not give you the value on the ballot.

Still, even if the webhooks does not tell the 3rd party App what the user voted, it still weakens the privacy model of Cocorico since even the information about who did (not) vote is not kept/made available in the platform itself as of today.

Another concern with webhooks is how unpredictable they are:

• a webhook could return a massive amount of data in order to crash/stall Cocorico
• a webhook could be unavailable when it is called by the platform
• any faulty webhook call could crash the corresponding requesting process and might prevent ballots to be fully processed
• we need to make sure the server behind the webhook is legit ("IP addresses can be spoofed, and DNS can be compromised, but fortunately we have cryptographic tools")

Webhooks best practices:

• Dropbox
• Mandrill
• Strip
• https://coffeeonthekeyboard.com/best-practices-for-happy-webhooks-1138/

Implementation ideas:

• webhook data should be a JWT signed with the destination App secret
• each webhook data should have a unique ID
• accept only HTTPS webhooks URIs
• webhook calls should be stored in a persistent queue and retried as long as they don't respond with 2XX
• webhook response should have a timeout (max 10sec for example)
• webhook queue handling should be a separate process

[thibauld]

Thanks for this spec. I agree. For implementation, what about adding that, in case of a webhook failure, the time between retrying should double after each failure (from 2s up to a certain limit, say 2048s (~68min)) and, if the webhook still fails to answer after 1 day, an email should be sent to the admin of the vote to let him know. As people can be lazy maintaining their webhook entry points, it would be safe to implement such protective measures. What do you think ?

[JMLX42]

what about adding that, in case of a webhook failure, the time between retrying should double after each failure (from 2s up to a certain limit, say 2048s (~68min))

Weirdly enough, one of the best practices link listed above actually states to do the opposite:

if the destination server isn’t available, the origin should retry it at increasing intervals

But it doesn't say why...

if the webhook still fails to answer after 1 day, an email should be sent to the admin of the vote to let him know

Yes: we should add an 'email' field to the App model.

[JMLX42]

Preliminary code written during the ODN4 hackathon is on the feature/webhooks branch.

[YoannAubineau]

I created a pull request in order to comment the current code: https://github.com/promethe42/cocorico/pull/31

[YoannAubineau]

a webhook could return a massive amount of data in order to crash/stall Cocorico

The worker could send HEAD requests so as to inform the remote server to trim the response body. However, nothing would prevent a malicious server to send a huge body regardless of what RFC 2616 says.

[JMLX42]

Replied to your comment on #31.

The worker could send HEAD requests so as to inform the remote server to trim the response body. However, nothing would prevent a malicious server to send a huge body regardless of what RFC 2616 says.

I hope the package we're using has an option to "enforce" an empty response. Meaning it will simply close the connection after receiving the headers.

Also keep in mind that - as of today - webhooks will be called on registered apps. And we register/validate those apps by hand. If one of those apps does anything fishy, we can just disable it.

[YoannAubineau]

@promethe42 Could you please detail what you have in mind regarding the whole webhoook workflow? As I understand, once a ballot transaction is written to the block chain, the contract triggers a Ballot event which is caught by the ballot queue worker in order to mark the ballot as 'complete' in MongoDB. I guess we should enqueue a webhook message right at that moment and let the webhook queue worker deal with it on its own, right?

[JMLX42]

we should enqueue a webhook message right at that moment and let the webhook queue worker deal with it on its own, right?

Yes.

We should add a "webhookURL" field to the App model and send the webhook event to that URL.

[YoannAubineau]

Ok, I think I've got a clear enough idea of what to do. Do you mind if I start working on it?

[JMLX42]

Ok, I think I've got a clear enough idea of what to do. Do you mind if I start working on it?

Let's go :) But we need a first working version ASAP since this is for LaPrimaire.org. When do you think you can delvier?

[YoannAubineau]

I am able to dedicate 1h of work every day of the week, which is about a working day per week. I should have the feature implemented by the end of next week I guess.

[JMLX42]

Great! thanks!