Notification webhook failures

[sshen81]

We are seeing webhook failures being reported for varying reasons (even when the overall build is successful).

https://app.shippable.com/bitbucket/vigorsystemsinc/service-antivirus/runs/295/1/console In this run, the build status was a success & the notification was sent to our webhook. However, we still got an error listed in the webhook step: Failed to trigger webhook for objectId:5e2b4aba4d96180007ab9186 with error:null

https://app.shippable.com/bitbucket/vigorsystemsinc/service-antivirus/runs/297/1/console In this run, the build status was a purposeful failure & the notification was not sent at all: Generated outPayload Failed to generated outPayload: Could not parse body

YAML:

  notifications:
    - integrationName: ms-teams-build-webhook
      type: webhook
      payload:
        - branch=$BRANCH
        - url=$BUILD_URL
        - build_number=$BUILD_NUMBER
        - commit=$COMMIT
        - compare=$COMPARE_URL
        - repo=$REPO_NAME
        - repo_full_name=$REPO_FULL_NAME
        - repo_url=$REPOSITORY_URL
        - success=$is_success
        - author=$COMMITTER
        - commit_message=$COMMIT_MESSAGE
      on_success: always
      on_failure: always
      on_start: never
      on_pull_request: never

[trriplejay]

In the first error, it looks like this can happen if the response body is empty. Do you know what response body you would typically get from your webhook endpoint? Regardless, it seems like a bug that an empty response body would cause a failure here.

The second error could happen if the build fails before all of the variables in the payload are defined. Since your injected failure is in pre-ci, I think this is likely the case. You could try forcing an error in the main ci section, something like cd baddir and see if you still get this issue. another thing to try would be to put quotes around all of the bash variables, like - commit="$COMMIT". This way, if the variables are undefined in the script environment, it will be interpreted as an empty string instead of a null value that causes a parsing error.

[sshen81]

Thanks for your response.

The webhook was returning only a status code without body. We can adjust it to return a body, but as you mention, it seems you should be handling null conditions.

Quotes on the variables results in the following error on successful builds. I don't think that's correct: https://app.shippable.com/bitbucket/vigorsystemsinc/service-antivirus/runs/299/1/console

Generated outPayload
Failed to generated outPayload: Could not parse body

[trriplejay]

ah, ok. you can remove the quotes then... I think if the job fails in the main CI section those variables should be filled in and the payload should be sent as expected. Pre-ci is a special case since much of the build environment hasn't been loaded yet.

[sshen81]

echo'd the env vars being used in the pre-ci step and they are all present. And removed all of their usage from the payload in the notification and it still fails with the same error Failed to generated outPayload: Could not parse body:

https://app.shippable.com/bitbucket/vigorsystemsinc/service-antivirus/runs/304/1/console

    - integrationName: ms-teams-build-webhook
      type: webhook
      payload:
        - title=Shippable webhook
      on_success: always
      on_failure: always
      on_start: never
      on_pull_request: never

It looks like there's no way to get a webhook to fire from failures before the ci step?? That's a major drawback and not at all like the behavior of other notification integrations. We have been using the slack notification integration and it doesn't have this limitation at all. Can you provide a fix for this?

[sshen81]

The intention behind the webhook was to allow that to send to our channel and let default shippable behavior send email. Since we haven't been able to get the webhook to work due to the above issues, we are trying to use the email delivery option since Teams allows email delivery to a channel.

However, attempts to use '--last_committer' or '- --last_committer' in the recipient list have failed to send.

[sshen81]

@trriplejay Wanted to see if you had any feedback or possible alternative options. Is webhook integration for failures outside of the ci step something that we can expect to be implemented?

[trriplejay]

unfortunately i don't think there's any other way to get a notification when the job fails in pre_ci

not sure why - --last_committer would not work though, is that specifically when it fails in pre_ci as well? pre_ci is a special case in a lot of ways.