search

tomasbjerre / pull-request-notifier-for-bitbucket

Bitbucket Server plugin that invokes a custom URL when a pull request event is triggered.
Other
185 stars 74 forks source link

Pull Request Notifier Plugin Blocking Threads on all https nodes #290

Open ankushguptajpmorgan opened 6 years ago

ankushguptajpmorgan commented 6 years ago

Hi Support Team

We are using the Pull Request Notifier Plugin on our Bitbucket (8 nodes cluster environment). We have 4 nodes dedicated for the https traffic and other 4 for the ssh traffic. We had production outage today on the https nodes on June 12 around 5 am EDT. We restarted our nodes in rolling fashion multiple times but it didn't help. It was only after disabling the plugin we had a positive result and the number of thread decreased immediately.

We were working with Atlassian for the RCA and found most of our threads were in the locked state.

t se.bjurr.prnfb.service.SettingsService$7.perform(SettingsService.java:318) at com.atlassian.stash.internal.user.DefaultEscalatedSecurityContext.call(DefaultEscalatedSecurityContext.java:51) at se.bjurr.prnfb.service.SettingsService.inSynchronizedTransaction(SettingsService.java:314)

i am attaching here thread dumps and the logs.

Atlassian also suspect that the plugin store data in the DB. This is probably how the Add-on was able to cause this problem after a reboot. We would like to know if there is a table with jobs in the DB that can be cleared out to allow you to use this Add-on again. We would like to get a complete RCA for this issue. Currently the plugin is in disabled state and lot of users are affected. Thanks,

Ankush Gupta JPMorgan Chase

tomasbjerre commented 6 years ago

Possibly related to: https://github.com/tomasbjerre/pull-request-notifier-for-bitbucket/issues/237

There is a PLUGIN_SETTING table. And in there you can do: SELECT * FROM PLUGIN_SETTING WHERE KEY_NAME LIKE '%pull%'; to find keys and SELECT KEY_VALUE FROM PLUGIN_SETTING WHERE KEY_NAME='se.bjurr.prnfb.pull-request-notifier-for-bitbucket-3' to find their value.

These are settings. Removing them would have the effect that the plugin will not have anything configured anymore.

A lot has happened since 2.63, if you look at the changelog: https://github.com/tomasbjerre/pull-request-notifier-for-bitbucket/blob/master/CHANGELOG.md

An upgrade of the plugin and Bitbucket Server is recommended!

Srikanthkakarla commented 6 years ago

Please suggest us the stable version to proceed with the plugin upgrade. Thanks

Regards, Srikanth Kakarla

tomasbjerre commented 6 years ago

Plugin version 3.22. And at least Bitbucket Server 5.1.0 but I would recommend latest 5.11.1.

ankushguptajpmorgan commented 6 years ago

We do have plans for the Bitbucket version upgrade in Q4 and it will have to go through extensive planning and testing. Can we upgrade the plugin to v 3.22 on our current Bitbucket v 4.12.1

tomasbjerre commented 6 years ago

No. 2.67 should work for you. Compatibility information here: https://marketplace.atlassian.com/apps/1213177/pull-request-notifier-for-bitbucket/version-history

Srikanthkakarla commented 6 years ago

So, We can upgrade the plugin to 2.67 on our current bitbucket v 4.12.1. Can we get the link to download the 2.67 plugin please.

tomasbjerre commented 6 years ago

It is in Atlassian Marketplace. The upgrade will not solve this issue. Check the changelog to see what the upgrade would add.

tomasbjerre commented 6 years ago

273