Bitbucket Server Catalog Plugin Event Support

[davelil4]

Bitbucket Server Catalog Plugin Event Support

I added event support for the Bitbucket Server catalog backend plugin, as well as the event backend plugin that is for that event support. I follow much of the same logic as the Bitbucket Server Cloud plugin with its event support. If I need to be more rigorous or clear with my testing please let me know.

:heavy_check_mark: Checklist

• [x] A changeset describing the change and affected packages. (more info)
• [x] Added or updated documentation
• [x] Tests for new functionality and regression tests for bug fixes
• [x] All your commits have a Signed-off-by line in the message. (more info)

[backstage-goalie[bot]]

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/plugin-catalog-backend-module-bitbucket-server	plugins/catalog-backend-module-bitbucket-server	minor	v0.1.33-next.1
@backstage/plugin-events-backend-module-bitbucket-server	plugins/events-backend-module-bitbucket-server	minor	v0.1.0

[github-actions[bot]]

Uffizzi Ephemeral Environment Deploying

:cloud: https://app.uffizzi.com/github.com/backstage/backstage/pull/19633

:gear: Updating now by workflow run 6502333863.

What is Uffizzi? Learn more!

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davelil4]

Hi, would you please be re-open the PR? I still wish to get this merged.

[bforbis]

@davelil4 I am beginning to test this PR now, as I'm very much interested in using this. A quick clarification as I begin testing, I ended up following the README instructions to install the plugins (copied the src as local internals for now), and I followed the events backend instructions to create packages/backend/src/plugins/events.ts and also wire it up to catalog.ts and index.ts.

I then copied a sample paload from atlassian documentation for the repo:refs_changed event and put that in postman

curl request

``` curl --location 'http://localhost:7007/api/events/http/bitbucketServer' \ --header 'X-Event-Key: repo:refs_changed' \ --header 'Content-Type: application/json' \ --data-raw '{ "eventKey": "repo:refs_changed", "date": "2023-01-13T22:26:25+1100", "actor": { "name": "admin", "emailAddress": "admin@example.com", "active": true, "displayName": "Administrator", "id": 2, "slug": "admin", "type": "NORMAL" }, "repository": { "slug": "rep_1", "id": 1, "name": "rep_1", "hierarchyId": "af05451fc6eb4bf4e0bd", "scmId": "git", "state": "AVAILABLE", "statusMessage": "Available", "forkable": true, "project": { "key": "PROJECT_1", "id": 1, "name": "Project 1", "description": "PROJECT_1", "public": false, "type": "NORMAL" }, "public": false, "archived": false }, "changes": [ { "ref": { "id": "refs/heads/master", "displayId": "master", "type": "BRANCH" }, "refId": "refs/heads/master", "fromHash": "197a3e0d2f9a2b3ed1c4fe5923d5dd701bee9fdd", "toHash": "a00945762949b7b787ecabc388c0e20b1b85f0b4", "type": "UPDATE" } ], "commits": [ { "id": "a00945762949b7b787ecabc388c0e20b1b85f0b4", "displayId": "a0094576294", "author": { "name": "Administrator", "emailAddress": "admin@example.com" }, "authorTimestamp": 1673403328000, "committer": { "name": "Administrator", "emailAddress": "admin@example.com" }, "committerTimestamp": 1673403328000, "message": "My commit message", "parents": [ { "id": "197a3e0d2f9a2b3ed1c4fe5923d5dd701bee9fdd", "displayId": "197a3e0d2f9" } ] } ], "toCommit": { "id": "a00945762949b7b787ecabc388c0e20b1b85f0b4", "displayId": "a0094576294", "author": { "name": "Administrator", "emailAddress": "admin@example.com" }, "authorTimestamp": 1673403328000, "committer": { "name": "Administrator", "emailAddress": "admin@example.com" }, "committerTimestamp": 1673403328000, "message": "My commit message", "parents": [ { "id": "197a3e0d2f9a2b3ed1c4fe5923d5dd701bee9fdd", "displayId": "197a3e0d2f9", "author": { "name": "Administrator", "emailAddress": "admin@example.com" }, "authorTimestamp": 1673403292000, "committer": { "name": "Administrator", "emailAddress": "admin@example.com" }, "committerTimestamp": 1673403292000, "message": "My commit message", "parents": [ { "id": "f870ce6bf6fe633e1a2bbe655970bde25535669f", "displayId": "f870ce6bf6f" } ] } ] } }' ```

When posting this from Postman, I get a successfull response from Postman but then the backstage processor fails with the following message:

2023-09-06T19:24:08.359Z backstage error Subscriber "BitbucketServerEntityProvider" failed to process event Cannot read properties of undefined (reading 'self') type=plugin stack=TypeError: Cannot read properties of undefined (reading 'self')
[1]     at BitbucketServerEntityProvider.getLocationEntity (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:262:45)
[1]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[1]     at async BitbucketServerEntityProvider.onRepoPush (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:297:25)
[1]     at async BitbucketServerEntityProvider.onEvent (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:278:13)
at async /Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:34:11
[1]     at async Promise.all (index 0)
[1]     at async DefaultEventBroker.publish (/Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:31:5)

Is this error expected? The error message doesn't seem to be too helpful, but I'm guessing it's possible this is because of the dummy data in the payload? I'm still working through some networking permissions to actually fully test this with bitbucket.

[bforbis]

After fussing with some WAF settings I was able to get this working in a full end-to-end test and happy to say it accepted the event for me with no errors! So this seems to work fine with legit payloads from bitbucket.

[davelil4]

I'm not sure why these other checks are failing but I adjusted the docs if you want to check @bforbis .

[bforbis]

@davelil4 I am beginning to test this PR now, as I'm very much interested in using this. A quick clarification as I begin testing, I ended up following the README instructions to install the plugins (copied the src as local internals for now), and I followed the events backend instructions to create packages/backend/src/plugins/events.ts and also wire it up to catalog.ts and index.ts.

I then copied a sample paload from atlassian documentation for the repo:refs_changed event and put that in postman

curl request When posting this from Postman, I get a successfull response from Postman but then the backstage processor fails with the following message:

2023-09-06T19:24:08.359Z backstage error Subscriber "BitbucketServerEntityProvider" failed to process event Cannot read properties of undefined (reading 'self') type=plugin stack=TypeError: Cannot read properties of undefined (reading 'self')
[1]     at BitbucketServerEntityProvider.getLocationEntity (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:262:45)
[1]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[1]     at async BitbucketServerEntityProvider.onRepoPush (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:297:25)
[1]     at async BitbucketServerEntityProvider.onEvent (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:278:13)
at async /Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:34:11
[1]     at async Promise.all (index 0)
[1]     at async DefaultEventBroker.publish (/Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:31:5)

Is this error expected? The error message doesn't seem to be too helpful, but I'm guessing it's possible this is because of the dummy data in the payload? I'm still working through some networking permissions to actually fully test this with bitbucket.

@davelil4 any idea if the error message here could be cleaned up to be more human readable?

[davelil4]

@davelil4 I am beginning to test this PR now, as I'm very much interested in using this. A quick clarification as I begin testing, I ended up following the README instructions to install the plugins (copied the src as local internals for now), and I followed the events backend instructions to create packages/backend/src/plugins/events.ts and also wire it up to catalog.ts and index.ts. I then copied a sample paload from atlassian documentation for the repo:refs_changed event and put that in postman curl request When posting this from Postman, I get a successfull response from Postman but then the backstage processor fails with the following message:

2023-09-06T19:24:08.359Z backstage error Subscriber "BitbucketServerEntityProvider" failed to process event Cannot read properties of undefined (reading 'self') type=plugin stack=TypeError: Cannot read properties of undefined (reading 'self')
[1]     at BitbucketServerEntityProvider.getLocationEntity (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:262:45)
[1]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[1]     at async BitbucketServerEntityProvider.onRepoPush (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:297:25)
[1]     at async BitbucketServerEntityProvider.onEvent (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:278:13)
at async /Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:34:11
[1]     at async Promise.all (index 0)
[1]     at async DefaultEventBroker.publish (/Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:31:5)

Is this error expected? The error message doesn't seem to be too helpful, but I'm guessing it's possible this is because of the dummy data in the payload? I'm still working through some networking permissions to actually fully test this with bitbucket.

@davelil4 any idea if the error message here could be cleaned up to be more human readable?

@bforbis I'll look into soon as I get the chance. Just to speed up time, what are the "WAF" settings you mentioned that you changed to fix the error? I had never previously gotten this error so it would help me out to know.

[bforbis]

@bforbis I'll look into soon as I get the chance. Just to speed up time, what are the "WAF" settings you mentioned that you changed to fix the error? I had never previously gotten this error so it would help me out to know.

Ignore the part about the WAF. That was just something blocking me from testing end-to-end with Bitbucket directly (ie: I needed to update my firewall to allow bitbucket to send requests to backstage).

The error that I saw before was from me testing the event webhook URL manually using Postman. Since I didn't yet have the bitbucket connection setup, I tried testing manually by using the sample payload provided on Bitbucket's doc site.

I am not surprised that this caused an error within backstage, as that sample payload is all dummy data. However, the error message reported by Backstage was very confusing and did not actually clue me into what was going wrong. I would have expected some sort of message like "Could not find associated entity", but instead we got some crazy JS undefined behavior error.

[davelil4]

@bforbis I'll look into soon as I get the chance. Just to speed up time, what are the "WAF" settings you mentioned that you changed to fix the error? I had never previously gotten this error so it would help me out to know.

Ignore the part about the WAF. That was just something blocking me from testing end-to-end with Bitbucket directly (ie: I needed to update my firewall to allow bitbucket to send requests to backstage).

The error that I saw before was from me testing the event webhook URL manually using Postman. Since I didn't yet have the bitbucket connection setup, I tried testing manually by using the sample payload provided on Bitbucket's doc site.

I am not surprised that this caused an error within backstage, as that sample payload is all dummy data. However, the error message reported by Backstage was very confusing and did not actually clue me into what was going wrong. I would have expected some sort of message like "Could not find associated entity", but instead we got some crazy JS undefined behavior error.

@bforbis I added some better logging. When you get the chance let me know how it looks.

[bforbis]

@bforbis I added some better logging. When you get the chance let me know how it looks.

Looks good. Following the same test plan by posting dummy data, I now get the following error log:

[1] 2023-09-15T22:20:05.732Z catalog error Failed to create location entity. type=plugin target=bitbucketServer-provider:bitbucketProvider

[bforbis]

@Rugvip requesting your review on this PR :)

[bforbis]

Worth mentioning, my company uses the Post Webhooks marketplace plugin for bitbucket. This plugin does similar things to native webhooks, but it is a lot more configurable, eg: you can cause it to only run on certain branches, ignore certain PR types, and most importantly can be configured globally across an entire bitbucket org, where-as native webhooks need to be configured for every project.

The problem is that they events are slightly different and won't work with the code that is being submitted here. They end up using repo:push event keys and the changes events show up in a slightly different structure.

Probably this will blow the scope of this PR to add it there, but I'm interested in this kind of support. It also addresses a few problems called out, like the ability to limit which branches events get consumed from (ie: this is configured in the webhook publisher itself).

[davelil4]

Going to get to more of these changes next week. I have some exams then I will try and work on some of the changes as they might take me a decent amount of time.

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[backstage-goalie[bot]]

Thanks for the contribution! All commits need to be DCO signed before they are reviewed. Please refer to the the DCO section in CONTRIBUTING.md or the DCO status for more info.

[freben]

Hi! To get this to green state:

• Sonsider squashing to a single commit - there's a fair bit of history and there are unnecessary conflicts along the way that the following step will hit.
• Rebase on a fresh master.
• Run yarn build:api-reports and make sure that they don't emit errors. You'll first get Error: The API Report for plugins/catalog-backend-module-bitbucket-server is not allowed to have warnings
• Commit the results of those api reports in the same PR.

then the builds should hopefully go green

[davelil4]

Hi! To get this to green state:

• Sonsider squashing to a single commit - there's a fair bit of history and there are unnecessary conflicts along the way that the following step will hit.
• Rebase on a fresh master.
• Run yarn build:api-reports and make sure that they don't emit errors. You'll first get Error: The API Report for plugins/catalog-backend-module-bitbucket-server is not allowed to have warnings
• Commit the results of those api reports in the same PR.

then the builds should hopefully go green

Hi! Just so I understand correctly. Do you want me to create a new PR from a fresh master branch with my one squashed commit at the top? Or do you want me to change this PR? Thank you!

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davelil4]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

Bumping to unmark stale. I implemented the default branch grab but need to write test cases for it.

[camilaibs]

• Sonsider squashing to a single commit

Hi, @davelil4 thank you so much for contributing to us and also for being so open to hearing the suggestions given by our colleagues that helped us to review your pull request 🙂

I think what @freben is saying is to keep this pull request and squash all of your commits to a single one, so it makes it easy for you to rebase this branch with the master.

[freben]

Indeed. Squashing isn't mandatory, but given the state of the branch I thought it might simplify things.

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davelil4]

Bumping to unmark stale.

[bforbis]

Hi @davelil4 just trying to keep track of where this PR is. Are there more things for me to review? Thank you so much for the work you've been putting into this!

[davelil4]

Hi @davelil4 just trying to keep track of where this PR is. Are there more things for me to review? Thank you so much for the work you've been putting into this!

Hi @bforbis, I am currently working on some of the other suggestions given, such as specific error reporting as asked of me from some previous suggestions. I am still working on the PR and recently got the Bitbucket Server trial so I have been testing. None of the other API endpoints in the client have the same sort of error logging so I have just been working on figuring it out and creating test cases. I will make sure to @ you when I push the next changes 😄 . I have finals coming up so I might not be able to complete these changes until two weeks from now but will do my best.

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davelil4]

Bumping to unmark stale, I will work on some finishing touches later this week/early next week. I will also rebase on fresh master to update properly.

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davelil4]

Working on rebasing from the updated master and going to try and get all my changes pushed by tomorrow.

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[github-actions[bot]]

Uffizzi Ephemeral Environment - Virtual Cluster

Your cluster pr-19633 was successfully created. Learn more about Uffizzi virtual clusters To connect to this cluster, follow these steps:

1. Download and install the Uffizzi CLI from https://docs.uffizzi.com/install
2. Login to Uffizzi, then select the backstage account and project:

   uffizzi login

Select an account: 
  ‣ backstage
    jdoe

Select a project or create a new project: 
  ‣ backstage-6783521

3. Update your kubeconfig: uffizzi cluster update-kubeconfig pr-19633 --kubeconfig=[PATH_TO_KUBECONFIG] After updating your kubeconfig, you can manage your cluster with kubectl, kustomize, helm, and other tools that use kubeconfig files: kubectl get namespace --kubeconfig [PATH_TO_KUBECONFIG]

Access the backstage endpoint at https://backstage-default-pr-19633-c5630.uclusters.app.uffizzi.com

[pjungermann]

I would like to bring PR #22344 to your attention as it might cause some changes

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davelil4]

I would like to bring PR #22344 to your attention as it might cause some changes

I will look into it.

[davelil4]

@bforbis I should've made all requested changes by now and it should work with your version of bitbucket server. I tested on my own instance and it was working, when you get the chance please try it out and see if there's anything else I can improve upon to get the pr done.

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[bforbis]

Still haven't had time to set down and look at this fully. I haven't forgotten!

[bforbis]

Hi @davelil4, I've tested this out and things seem to be working well

[Edje-C]

@pjungermann can you re-take a look at this?

[davelil4]

Should I rebase my changes onto the latest master branch or will this remove the recent approvals?

[Edje-C]

Should I rebase my changes onto the latest master branch or will this remove the recent approvals?

It won't remove approvals, you're good 👍🏾

[davelil4]

I would like to bring PR #22344 to your attention as it might cause some changes

In this next commit I accounted for this change now that that branch is merged.

[davelil4]

@Rugvip I fixed the yarn.lock issues, all my merge checks should be passing if you can please review the changes you requested.

[davelil4]

I would suggest switching to the new EventsService implementation. Of course, this is also possible as following PR. However, adopters of it would need to adjust again.

Thank you for letting me know, I will get to fixing all of this the next week or two.

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davelil4]

Still working.