Webhook stage, integrate against anything

[amanya]

Create a new stage for calling an external webhook / service, that will be useful for having a flexible way of interacting with technologies that have not yet been integrated with Spinnaker, for example running AWS lambdas or running Marathon deployments.

Description

We'll add a new pipeline stage type that will call a user-defined endpoint, passing to it a json document with fields customized in the stage's ui using pipeline expressions.

When configuring the stage, we will have a checkbox to "wait for results". If it is not checked, the stage will be succeeded if the webhook's status code is 2xx or failed otherwise.

If "wait for results" is checked, Spinnaker will expect a 2xx return code along with a callback url that can be returned in the "Location" header or in the body of the response. This url will be used to get the status of the task we triggered, it will return a JSON like:

{
    "status": "[NOT_STARTED|RUNNING|SUCCEEDED|TERMINAL|...]",
    "info": "Some information about the process"
}

This information will be used to update the status of the execution and give feedback to the user.

To give this the maximum flexibility, there will be a configurable (JSONPath notation perhaps) way of specifying what response element to look at, and what values map to success/failure.

As a next step, it will be nice to have config-driven definitions of external services that would then be presented as choices when using this new stage.

Implementation details

We decided that the best fit for putting this is Orca, because it has similar tasks as the one we want to implement, for example in the bake stage it triggers a task with parameters, waits for it to complete and update the execution accordingly. Also the jenkins and run script stages are good examples of this.

There has been some discussion about this feature before opening this issue in this google document.

[amanya]

This is a screenshot of the inital works in the stage's configuration view:

[robzienert]

Looks like great progress; I know it's still in the works, but would like to see the textbox for "Status URL path" to be hidden, and slotted below its label when visible.

[amanya]

TODO list for next iterations:

• Define new custom stages based on the webhook stage by configuring them in orca.yml. Those will be selectable in the ui as if they were regular stages.
• "try it out" button that will show the results of the webhook call next to it so that the json path values could be tested. Similar to to how expresssions evaluate to results.
• Beautify the payload in the execution view. Limit the number of lines to be shown with an option to show the rest (clamp)
• Store a log of all calls to the status endpoint and show a link to view it in the same way as the bake log.
• Pass custom headers to the webhook call

[robzienert]

Super exciting todo list!

On Mar 31, 2017 4:44 AM, "Albert Manyà" notifications@github.com wrote:

TODO list for next iterations:

• Define new custom stages based on the webhook stage by configuring them in orca.yml. Those will be selectable in the ui as if they were regular stages.
• "try it out" button that will show the results of the webhook call next to it so that the json path values could be tested. Similar to to how expresssions evaluate to results.
• Beautify the payload in the execution view. Limit the number of lines to be shown with an option to show the rest (clamp)
• Store a log of all calls to the status endpoint and show a link to view it in the same way as the bake log.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/spinnaker/spinnaker/issues/1512#issuecomment-290690880, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGoxZleKEuaGV78Bff2LsxljkTtPyAoks5rrOcWgaJpZM4MqF3o .

[Nagarajj]

This would be helpful, would there be a way to pass headers to the web service url. It seems to assume no headers to be present.

@Service class WebhookService {

@Autowired RestTemplate restTemplate

ResponseEntity

[amanya]

@Nagarajj this is a feature other people requested, we'll probably implement it soon :)

[Nagarajj]

@amanya I have made changes to addHeaders to WebService requests. Have submitted a PR, please see if that makes sense.

• https://github.com/spinnaker/deck/pull/3596
• https://github.com/spinnaker/orca/pull/1302

1. Headers are added to both the webService calls. Headers could be things such as "Authorization" headers for authentication (required by both "execute" and "getStatus" requests).
2. Added support for JSONPath returning list response such as selecting last element in array "$.[-1:].status" or find running element in response "$.[?(@.status == "RUNNING")].status"
3. Fixed a minor existing UI bug where default "statusUrlResolution" of "getMethod" was not set as it was not initialized.

Below is the screenshot of change

[amanya]

@Nagarajj Ups, we already did something very similar: https://github.com/spinnaker/deck/pull/3590

Would you wait for our PR to be merged and then add your changes in a new one? Or do you think it's worth updating ours? Your code seems much simpler but I can't look at in in depth right now.

[Nagarajj]

@amanya sure, will wait for your PR to be merged. One Q ? will the headers support both "execute" and "getStatus" webservice calls. Would be helpful if it supports both. Rest of the issues are simpler and will raise a seperate PR for those. Thanks.

[amanya]

@Nagarajj No, we only implemented for the execute call, feel free to add it for the getStatus as well, good idea!

[Nagarajj]

@amanya, should we share the headers across both the calls or provide ability to define separate headers for the getStatus call.

The use case we are trying to solve for, ability to define "Authorization" header which is required for both execute and getStatus call. Not sure of an use case where a separate set of headers would be required across execute and getStatus.

Let know your thoughts and i will get going on thins.

[erran]

Is this issue still on going after the Webhook stage and predefined webhook stages after spinnaker/orca#1329?

[amanya]

@erran no, I think we can close it, thanks for the heads up!