Add Support for BitBucket webhooks

[bkmeneguello]

ISSUE TYPE

• Feature Request

SUMMARY

Restore commit 885841caeaea0f7c86d39d6e7a66b960124289c2 to add support for BitBucket webhooks, since Server edition supports signed payloads and Cloud edition suggests IP Whitelists (manage webhooks and ip whitelist)

@jbradberry

[rmauc]

+1 for Bitbucket Server

[nitinkansal1984]

+1 for bitbucket

[Forest-Troll]

I try restore this commit with some changes. It fine work with bitbucket 5.12 and AWX 9.1.0 for me. Not tested this with last environment yet.

https://github.com/Forest-Troll/awx/tree/bitbucket-return

[ryanpetrello]

Support for this in AWX is blocked, because Bitbucket does not support signed webhooks in its free product. Details at:

https://github.com/ansible/awx/pull/8328#issuecomment-707171773

[christianmarangoni]

+1

[rmauc]

Is there any update on this RFE? We are looking for the webhook to Bitbucket Server (on-prem) V7.6.0 Thanks

[jvandermeulen]

+1 for Bitbucket Server

[jbradberry]

@rmauc @jvandermeulen https://github.com/ansible/awx/pull/8328#issuecomment-707171773 as noted above.

[rmauc]

Is waiting for the free product really justification to restrict enterprise customers from fully benefiting from the Tower feature set? Afterall, you can get 10 x Bitbucket Server licenses for $10USD per year. https://www.atlassian.com/licensing/bitbucket#server-licensing

[opfpqgoon]

Why could we not enable bitbucket on-prem (server/datacenter) Webhooks with clear documentation & UI tooltips explaining that this does not work with bitbucket cloud?

Having bitbucket Webhooks integration with AWX would massively simplify CI/CD workflows as there's no longer a need for an intermediary service to receive Webhooks and trigger tower-cli commands.

I'd be happy to help test anything related to bitbucket on-prem webhook integration.

[al-lac]

Please consider providing this for the on-premise server / datacenter versions. I know a lot of companies that would benefit from this integration.

[tech2734]

I am running into more and more clients that are using Bitbucket and have a pretty clear need for this functionality. Please reconsider supporting it. I would help out with testing as well. thanks.

[derifgig]

+1 for Bitbucket

[vinealvees]

+1 for bitbucket

[pkostadinov90]

This is really useful feature. We really need it!

However there seems to be some changes and end of life for the old Server edition bitbucket. So you may no longer need to support 2 different bitbucket workflows, which would make adding back this feature a bit easier.

[lucab85]

+1

[nacurtis]

+1

[Houssem-Azzouz]

+1 for bitbucket webhooks

[Nothing4You]

please stop commenting +1.

it really just annoys everyone that's watching this issue/repo as they all get an email every time. you can add reactions to the initial post to indicate your interest.

[philipsd6]

Support for this in AWX is blocked, because Bitbucket does not support signed webhooks in its free product. Details at:

#8328 (comment)

@ryanpetrello This would sound more reasonable if AWX didn't support GitLab, which also does not support signed webhooks. As of today, GitlabWebhookReceiver() simply compares the HTTP_X_GITLAB_TOKEN header value (clear text!) with the webhook_key. This is doable with Bitbucket.org too.

[ryanpetrello]

Hey @philipsd6,

I'm less involved with this project than I used to be, and am no longer an active contributor. @shanemcd @AlanCoding @chrismeyersfsu ☝️

[philipsd6]

Good to know! Thanks for all your contributions and support in the past!

[shanemcd]

Just an update for folks. We are not ignoring your requests here, but we will not be adding this until Bitbucket handles the underlying issue on their end. I encourage you to weigh in on these related issues:

• https://jira.atlassian.com/browse/BCLOUD-14683
• https://jira.atlassian.com/browse/BCLOUD-12195

[al-lac]

@shanemcd but what about Bitbucket Server and Datacenter? Support for Webhook Secrets is already implemented there. And it will probably take some time for atlassian to implement it in the cloud version. 😕

[shanemcd]

We do not have a budget for purchasing licenses to test and support integrations with enterprise software. Maybe if Atlassian wanted to hook us up we could consider it? But I find it morally questionable they chose to withhold a critical security feature from their cloud-based offering.

[philipsd6]

Atlassian has (currently) only chosen to support IP allow lists, so at least they are supporting reducing the attack surface. I and others have encouraged Atlassian to support signed webhooks. But @shanemcd, I respectfully disagree with the decision here for BitBucket, for the following reasons:

1. BitBucket Server/DataCenter supports them -- and you can rely on me (and likely others here) to implement and test the feature.

2. As noted in my comment above, AWX currently supports GitLab, which does not support signed webhooks either:

   # GitLab only returns the secret token, not an hmac hash.

My proposal is that we/I add a BitBucketWebhookReceiver class that transparently handles the signed webhooks from BitBucket Server/DataCenter and falls back on the GitLab method of comparing the webhook key to a token added to the webhook url in BitBucket Cloud. (A fatal error with an instructive message if no token is found, and yes, less secure because the token is in the URL.) But we could even possibly get the client IP from the request object (HTTP_X_FORWARDED_FOR or REMOTE_ADDR headers) and ensure it matches the documented BitBucket allowlist. It's not ideal, admittedly, but there are other ways Ansible/AWX allows users to be insecure by choice, and I would rather see this project doing the best with what's available, and just provide warnings when less secure paths are taken.

FWIW, the allowlist can be retrieved (using HTTPie and Jello):

http GET https://ip-ranges.atlassian.com/ \
    | jello -lr '[item["cidr"] for item in _["items"]
                  if "bitbucket" in item.get("product", "")
                  and item["direction"] == ["egress"]
                  and item["mask_len"] < 32]'

[shanemcd]

@philipsd6 Well thank you for being respectful about it. 😄 I am not sure how I missed your comment above, I apologize. I just looked in the bitbucket cloud documentation and do not see anything in the payload similar to HTTP_X_GITLAB_TOKEN. Did I miss something?

If you are willing to send a PR that implements your proposal we'll give it an honest review and consideration. We also have someone who has reached out to Atlassian to see if they'd be willing to give us a license for testing.

[philipsd6]

Did I miss something?

No @shanemcd, you definitely didn't miss anything! What I was saying was that one could have the webhook url be https://awx.mysite.net/api/v2/job_templates/1/bitbucket/?token=b0e37dd0-b759-11ec-b14b-bb12b88566fd and then AWX would compare that token to the webhook key, same as it does for GitLab. That's obviously less user-friendly than we would like, but AFAIK is equally as secure as doing the same with the GitLab HTTP_X_GITLAB_TOKEN header.

It does imply that perhaps there should be two Webhook Services listed: BitBucket Cloud (for the autogenerated url to include the token) and BitBucket DataCenter to just use the signed webhook method. Explicit being better than implicit...

I am willing to submit a PR but my company workload is currently very high. I'll try to get a draft PR going next week.

[ibuchanan]

@philipsd6 As an Atlassian, I second your suggestion (implication) that there should be 2 options, separating Bitbucket Cloud from Bitbucket Data Center. What may not be obvious from the branding is these are 2 very different products, maintained by 2 different Atlassian teams. There's only informal connection between the features of one to the other. Hence, the shape of webhooks are quite different between the 2 products and it would be worth treating the 2 as if they were different products.

It looks like you have a possible path forward for Bitbucket Data Center, which seems the more popular need based on comments here. I wonder if it would be worth renaming this issue for Bitbucket Data Center and opening a new issue for Bitbucket Cloud?

For Bitbucket Cloud, there is a JWT-based webhook authentication mechanism when using Bitbucket Connect. The Bitbucket Connect architecture would have considerable overhead for AWX but, if someone does have ideas how it could work, feel free to at-mention me for input. Given the UI-based webhooks would be an easier means of integration, I have commented privately on the issues to prompt action.

[brunogomes011]

+1 for BitBucket

[dantecl]

Howdy folks,

This PR has been open for quite a while, and it's significant functionality. Any chance this could be revisited soon? I hate having to involve a jenkins instance to do what bitbucket server could do directly with AWX.

[Scriptiee]

To any one impatient to wait for this implementation, I managed to have bitbucket webhooks working by utilising nginx's "proxy_set_header" that sits in front of AWX.

It's not an ideal solution but allows you to send the payload to awx job for processing.

The gist of it is:

Enable GitLab webhook which will give you a API link as well as token for authentication.

On nginx add, customised to your set up of course:

location ~ [jobid]/gitlab/ { proxy_pass http://awx; proxy_set_header Host $host;

below is the line that handles the authentication

proxy_set_header x-gitlab-token [token-created-by-awx-for-that-particular-job];

}

And Bob's your uncle. The payload is sent to AWX as an extra var under tower_webhook_payload.

The nginx directives will need to be added for each job that requires webhook capabilities. It's a bit of a hack I guess but been nearly 4 years since this request was made.