search

reconquest / atlassian-external-hooks

External Hooks plugin for Atlassian Bitbucket
https://external-hooks.reconquest.io
Other
44 stars 37 forks source link

Add compatibility with Bitbucket Server >= 6.0 #81

Closed sebastic closed 5 years ago

sebastic commented 5 years ago

Bitbucket Server 6.0.0 has been released, and the external hooks addon is not yet compatible.

Please add compatibility with Bitbucket Server 6.x.

bturner commented 5 years ago

I'm the principal developer for Bitbucket Server. Before anyone spends too much time trying to do this, I just want to note that it is not possible to make External Hooks compatible with Bitbucket Server 6.0.

External Hooks relies on the ability to execute arbitrary processes in a repository's directory. The APIs which provided access to repository directories on disk were all removed without replacement in Bitbucket Server 6.0. That includes methods like:

As a result, it's no longer possible for External Hooks to run any process other than git in a repository's directory. (Note that this was communicated in advance via our API changelog when Bitbucket Server 5.10 was released, 9+ months ago.)

Stay tuned for further updates from Atlassian on this issue. While direct access to repositories on disk will never be restored in any future release, we're actively working on some new Java APIs to facilitate apps like External Hooks. (And by "we" I mean me.)

Best regards, Bryan Turner Atlassian Bitbucket

sebastic commented 5 years ago

Thanks for the info @bturner, I look forward to updates that will allow us to run git-multimail via External Hooks on Bitbucket Server >= 6.0.

nktl commented 5 years ago

Looking forward to some development here, the plugin stopped working post 6.x upgrade - extremely disappointing :(

kovetskiy commented 5 years ago

@nktl hey, don't worry, we already evaluated ways to solve the problem and the development is already in progress. We expect to get some updates on it on 21 March.

sleavitt commented 5 years ago

Just wondering if there were any updates on this since it is now 22 March :)

seletskiy commented 5 years ago

@sleavitt Unfortunately, it takes more time than expected, but implementation is in progress and hopefully there will be some solution on next week.

bturner commented 5 years ago

A new API for allowing apps to create hook scripts has been merged for inclusion in Bitbucket Server 6.2.0, which is likely to be released at some point next week. (I didn't want to jump the gun on providing a further update until the new API was fully merged and then potentially have to backtrack if everything wasn't done in time.)

With this new API, Bitbucket Server will manage the persistence of hook scripts, as well as executing them and either forwarding their output to clients (for pushes) or adding vetos (for other operations like branch/tag creation/deletion, pull request merges, etc.). App developers can configure the specific trigger or triggers their hook scripts are executed for, or enable them for all triggers, and that configuration can be applied globally, per-project or per-repository. Bitbucket Server also handles the differences between invoking hook scripts on Windows vs. Linux or macOS. (Note that even on Windows hook scripts are run in a POSIX-like environment, similar to the environment they would get if they were written directly into a Git repository as a pre-receive or post-receive hook. Our intention is to provide an environment that is, as much as is feasible, consistent across operating systems and, importantly, consistent with Git.) Hook scripts can be written in any language, and can use #! directives to run the "correct" interpreter.

I'm not sure what implementation(s) may have been explored for 6.x compatibility, but as someone intimately familiar with Bitbucket Server's Java API I know that there is no possible way External Hooks could be made compatible with 6.0 or 6.1 without violating our API boundaries to get direct access to repositories on disk. I would strongly encourage that any such approach be abandoned, and instead that you wait for 6.2.0 to be released (at which time developer documentation, and API documentation for the new HookScriptService, will be made public), and investigate updating External Hooks to leverage the new API. (I can't force you to do so, of course, and I wouldn't try. But I'd strongly encourage it. Note that any app that bypasses Bitbucket Server's API to access repositories on disk cannot be Data Center certified, and if we find such behavior in a certified app its certification will be revoked.) Using the new HookScriptService API will not allow External Hooks to be compatible with 6.0.x or 6.1.x, but 6.2.x and newer will be possible.

Best regards, Bryan Turner Atlassian Bitbucket

seletskiy commented 5 years ago

@bturner: Thanks for valuable information. Does that mean, that starting from Bitbucket 6.2.0 users will be able to directly specify repository hooks in Bitbucket UI without using any third-party plugins, like this one?

bturner commented 5 years ago

@seletskiy

The new hook scripts API has been added specifically to support app developers, and is Java API only. There are no REST endpoints or UI screens to allow users or administrators to create or configure hook scripts. While I can't say that those will never be added (I'm reasonably confident we'll get a feature request for those pretty quickly after 6.2.0 is released), at least for the near term there aren't any plans to do so. It will require a plugin, whether internal (to the organization) or third-party, to be able to use hook scripts.

One thing I'll add is that, if/when Bitbucket Server does add REST and UI treatments for hook scripts into the base product, only SYS_ADMIN users will be able to create, view, update or delete hook scripts. Since hook scripts basically function as remote-code-execution-as-a-service, only sysadmins, who have unrestricted access to the system, should be able to manage them. Once one or more hook scripts have been created, the system (again, if/when REST and UI treatments ship) would allow ADMIN users to configure them to run globally, PROJECT_ADMINs to run for all repositories in a project, or REPO_ADMINs to run for a specific repository. In other words, "lesser" admins would be able to select from a list of pre-existing (and therefore pre-vetted and pre-approved) hook scripts as appropriate for their needs, but would not be able to create their own. If a "lesser" admin needs a hook script for a particular purpose, they would need to work with a full sysadmin to get it created, and then they could configure it.

The HookScriptService as it's shipping in 6.2.0 enforces exactly those permission checks (this is all described in its Javadocs, I'll add; consider this a preview). Apps can, of course, use the SecurityService to bypass those permission checks, according to their own judgment and based on their use case.

So: No, in Bitbucket Server 6.2.0 users will not be able to directly specify hook scripts. It will require the use of an app to provide any user-visible access to hook scripts.

Best regards, Bryan Turner Atlassian Bitbucket

seletskiy commented 5 years ago

@bturner: Thanks for comprehensive reply, really appreciate it.

We will wait till 6.2.0 version then and will not release 6+ compat right now.

nktl commented 5 years ago

Hey, 6.2.0 is out now, just wondering if the API made it? (nothing mentioned in the changelog about it)

bturner commented 5 years ago

@nktl,

It made it. I wouldn't have commented here about it if it wasn't a sure thing. It looks like the developer who wrote the documentation overlooked adding an API changelog entry. I'll get that addressed.

In the interim, developer documentation is here:

One important point is that app developers are expected/required to track the HookScript.getId() of hook scripts they create so that they can manage those scripts later. Hook script names are not required to be unique (for an app like External Hooks, what would they use for a name given they're configured ad hoc?), so there's no mechanism for looking hook scripts up by name.

Best regards, Bryan Turner Atlassian Bitbucket

seletskiy commented 5 years ago

Just want to mention that work already in progress for External Hooks to be compatible with 6.2.0 using new APIs.

georgy commented 5 years ago

@bturner I think I found a bug in the implementation of the hook scripts (or I might be holding it wrong, would not be the first time). While working on https://github.com/reconquest/atlassian-external-hooks/pull/85 I saw this behavior:

sebastic commented 5 years ago

Is there an ETA for the external-hooks addon release with Bitbucket 6.x support?

We need to upgrade to 6.x for Java 11 support.

dploeger commented 5 years ago

This issue is actually closed. The release is already there. Check out: https://marketplace.atlassian.com/apps/1211631/external-hooks/version-history

dploeger commented 5 years ago

It's compatible since version 6.2.0-1. However, the API has changed and I also have to check, if the script running on our systems still does what it's supposed to. 😀

sebastic commented 5 years ago

After upgrading our test instance to 6.5.1 and upgrading the addon, enabling it fails. In the logging we see:

2019-08-21 08:20:38,629 ERROR [ThreadPoolAsyncTaskExecutor::Thread 58]  c.a.plugin.osgi.factory.OsgiPlugin Unable to start the plugin container for plugin 'com.ngs.stash.externalhooks.external-hooks'
org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from URL [bundle://130.0:0/META-INF/spring/atlassian-plugins-component-imports.xml]; nested exception is java.lang.IllegalStateException: The bundle is uninstalled.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:419)
        at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:171)
        at org.eclipse.gemini.blueprint.context.support.OsgiBundleXmlApplicationContext.loadBeanDefinitions(OsgiBundleXmlApplicationContext.java:141)
        at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:133)
        at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:636)
        at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$800(AbstractDelegatedExecutionApplicationContext.java:57)
        at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$3.run(AbstractDelegatedExecutionApplicationContext.java:239)
        at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
        at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.startRefresh(AbstractDelegatedExecutionApplicationContext.java:217)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.stageOne(DependencyWaiterApplicationContextExecutor.java:224)
        at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.refresh(DependencyWaiterApplicationContextExecutor.java:177)
        at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.refresh(AbstractDelegatedExecutionApplicationContext.java:154)
        at org.eclipse.gemini.blueprint.extender.internal.activator.LifecycleManager$1.run(LifecycleManager.java:213)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.lang.Thread.run(Thread.java:748)
        ... 6 frames trimmed
Caused by: java.lang.IllegalStateException: The bundle is uninstalled.
        at org.apache.felix.framework.Felix.getBundleResources(Felix.java:1706)
        at org.eclipse.gemini.blueprint.util.BundleDelegatingClassLoader.getResources(BundleDelegatingClassLoader.java:186)
        at org.springframework.core.io.support.PropertiesLoaderUtils.loadAllProperties(PropertiesLoaderUtils.java:178)
        at org.eclipse.gemini.blueprint.extender.internal.support.NamespacePlugins$Plugin.resolve(NamespacePlugins.java:77)
        at org.eclipse.gemini.blueprint.extender.internal.support.NamespacePlugins$5.operate(NamespacePlugins.java:209)
        at org.eclipse.gemini.blueprint.extender.internal.support.NamespacePlugins$5.operate(NamespacePlugins.java:205)
        at org.eclipse.gemini.blueprint.extender.internal.support.LazyBundleRegistry.apply(LazyBundleRegistry.java:159)
        at org.eclipse.gemini.blueprint.extender.internal.support.NamespacePlugins.doResolve(NamespacePlugins.java:205)
        at org.eclipse.gemini.blueprint.extender.internal.support.NamespacePlugins.resolve(NamespacePlugins.java:169)
        at org.eclipse.gemini.blueprint.context.support.TrackingUtil$OsgiServiceHandler.invoke(TrackingUtil.java:106)
        at org.eclipse.gemini.blueprint.context.support.DelegatedNamespaceHandlerResolver.resolve(DelegatedNamespaceHandlerResolver.java:55)
        ... 15 common frames omitted

Stopping the service, removing $BITBUCKET_HOME/plugins/.osgi-cache/, and starting the service again resolved the issue.

sebastic commented 5 years ago

Pushing a commit to the test repo triggers the wrapper hook for git-multimail which sends the commit diff as expected.