Argelbargel / gitlab-branch-source-plugin

Jenkins-Plugin to create a multi-branch-project from gitlab
GNU General Public License v2.0
71 stars 44 forks source link

Listeners seem to not be added correctly #25

Closed ananace closed 7 years ago

ananace commented 7 years ago

Ran into a new issue with our production Jenkins today where no listeners ended up being registered. This led to issues with all GitLab webhook calls resulting in 404 responses.

Jenkins version is 2.67 And if it helps, plugins and versions;

bouncycastle-api 2.16.1, structs 1.9, credentials 2.1.14, workflow-step-api 2.11, scm-api 2.1.1, workflow-api 2.17, script-security 1.29, workflow-support 2.14, durable-task 1.14, workflow-durable-task-step 2.12, swarm 3.4, display-url-api 2.0, mailer 1.20, ldap 1.15, workflow-scm-step 2.6, ace-editor 1.1, jquery-detached 1.2.1, workflow-cps 2.36, plain-credentials 1.4, pipeline-input-step 2.7, ssh-credentials 1.13, timestamper 1.8.8, git-client 2.4.6, junit 1.20, matrix-project 1.11, git 3.3.1, gitlab-plugin 1.4.5, envinject-api 1.2, envinject 2.1.2, workflow-job 2.13, pipeline-stage-step 2.2, pipeline-graph-analysis 1.4, pipeline-rest-api 2.8, handlebars 1.1.1, momentjs 1.1.1, pipeline-stage-view 2.8, pipeline-build-step 2.5, windows-slaves 1.3.1, external-monitor-job 1.7, cloudbees-folder 6.0.4, git-server 1.7, workflow-cps-global-lib 2.8, credentials-binding 1.12, branch-api 2.0.10, workflow-basic-steps 2.5, workflow-multibranch 2.16, icon-shim 2.0.3, authentication-tokens 1.3, docker-commons 1.7, docker-workflow 1.12, pipeline-model-api 1.1.7, pipeline-model-extensions 1.1.7, pipeline-model-declarative-agent 1.1.1, pipeline-stage-tags-metadata 1.1.7, pipeline-model-definition 1.1.7, pipeline-milestone-step 1.3.1, workflow-aggregator 2.5, matrix-auth 1.6, resource-disposer 0.6, ws-cleanup 0.33, javadoc 1.4, maven-plugin 2.16, ant 1.5, analysis-core 1.87, warnings 4.62, role-strategy 2.5.0, ownership 0.10.0, pam-auth 1.3, antisamy-markup-formatter 1.5, simple-theme-plugin 0.3, log-parser 2.0, saml 0.14, kubernetes 0.11, ldapemail 0.8, build-environment 1.6, gitlab-branch-source 0.6.4

Managed to debug the symptom down to;

def wh = argelbargel.jenkins.plugins.gitlab_branch_source.hooks.GitLabSCMWebHook.get()
def man = wh.manager

println man.managedListeners
// "[:]"

Ended up manually adding a listener for the root as a temporary solution;

def wh = argelbargel.jenkins.plugins.gitlab_branch_source.hooks.GitLabSCMWebHook.get()
def man = wh.manager

def root = Jenkins.instance.getItem('GitLab')
wh.addListener(root.navigators[0], root)

println man.managedListeners
// "[525018ff09dde899935859cac6d53bbf:argelbargel.jenkins.plugins.gitlab_branch_source.hooks.HookManager$ListenerState@4a8cf299]"

No idea about the cause, and the startup log doesn't give much information either;

Jun 29, 2017 10:37:59 AM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @577ms to org.eclipse.jetty.util.log.JavaUtilLog
Jun 29, 2017 10:37:59 AM winstone.Logger logInternal
INFO: Beginning extraction from war file
Jun 29, 2017 10:37:59 AM org.eclipse.jetty.server.handler.ContextHandler setContextPath
WARNING: Empty contextPath
Jun 29, 2017 10:37:59 AM org.eclipse.jetty.server.Server doStart
INFO: jetty-9.4.z-SNAPSHOT
Jun 29, 2017 10:38:00 AM org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet
INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Jun 29, 2017 10:38:00 AM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: DefaultSessionIdManager workerName=node0
Jun 29, 2017 10:38:00 AM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: No SessionScavenger set, using defaults
Jun 29, 2017 10:38:00 AM org.eclipse.jetty.server.session.HouseKeeper startScavenging
INFO: Scavenging every 660000ms
Jenkins home directory: /var/lib/jenkins found at: SystemProperties.getProperty("JENKINS_HOME")
Jun 29, 2017 10:38:00 AM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started w.@68c9d179{/,file:///var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/war}
Jun 29, 2017 10:38:00 AM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@1be2019a{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
Jun 29, 2017 10:38:00 AM org.eclipse.jetty.server.Server doStart
INFO: Started @1587ms
Jun 29, 2017 10:38:00 AM winstone.Logger logInternal
INFO: Winstone Servlet Engine v4.0 running: controlPort=disabled
Jun 29, 2017 10:38:01 AM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Jun 29, 2017 10:38:01 AM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Jun 29, 2017 10:38:06 AM jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
Jun 29, 2017 10:38:06 AM jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
Jun 29, 2017 10:38:07 AM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error
INFO: Failed to instantiate optional component org.csanchez.jenkins.plugins.kubernetes.ServiceAccountCredential$DescriptorImpl; skipping
Jun 29, 2017 10:38:07 AM jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
Jun 29, 2017 10:38:11 AM jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
Jun 29, 2017 10:38:11 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Download metadata
Jun 29, 2017 10:38:11 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 4 ms
Jun 29, 2017 10:38:11 AM jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
Jun 29, 2017 10:38:11 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@50548d20: display name [Root WebApplicationContext]; startup date [Thu Jun 29 10:38:11 CEST 2017]; root of context hierarchy
Jun 29, 2017 10:38:11 AM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@50548d20]: org.springframework.beans.factory.support.DefaultListableBeanFactory@d1fabab
Jun 29, 2017 10:38:11 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@d1fabab: defining beans [filter,legacy]; root of factory hierarchy
Jun 29, 2017 10:38:11 AM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
Jun 29, 2017 10:38:18 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #1 from /10.133.0.13:35010
Jun 29, 2017 10:38:18 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #3 from /10.133.0.7:39590
Jun 29, 2017 10:38:18 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #4 from /10.133.0.8:50444
Jun 29, 2017 10:38:18 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #2 from /10.133.0.9:58456
Jun 29, 2017 10:38:18 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #7 from /10.133.0.11:40100
Jun 29, 2017 10:38:18 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #6 from /10.133.0.10:47168
Jun 29, 2017 10:38:18 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #5 from /10.133.0.11:46052
Jun 29, 2017 10:38:23 AM org.eclipse.jetty.server.handler.ContextHandler$Context log
INFO: While serving https://jenkins.it.liu.se/script: hudson.security.AccessDeniedException2: anonymous is missing the Overall/RunScripts permission
Jun 29, 2017 10:38:26 AM org.opensaml.common.binding.security.SAMLProtocolMessageXMLSignatureSecurityPolicyRule evaluate
INFO: SAML protocol message was not signed, skipping XML signature processing
Jun 29, 2017 10:38:27 AM org.apache.xml.security.signature.Reference verify
INFO: Verification successful for URI "#_df8e6151-d8ff-46ec-86f9-7e1e41a0fb3e"
Jun 29, 2017 10:38:27 AM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error
INFO: Failed to instantiate optional component org.csanchez.jenkins.plugins.kubernetes.ServiceAccountCredential$DescriptorImpl; skipping
Jun 29, 2017 10:38:44 AM jenkins.branch.MultiBranchProject$BranchIndexing run
INFO: GitLab/group%2Fproject #20170629.103843 branch indexing action completed: SUCCESS in 1.5 sec

(Final messages are from me logging in and manually launching a project scan.)

Hope you're not sick of all my issues yet.

Argelbargel commented 7 years ago

On the contrary. I'm really glad to have someone testing out all the little things that i've missed ;-) I'm just swamped with work right now so i am not sure when i'll get to look into it. Are you using the current 0.7-dev build or the older 0.6.4?

ananace commented 7 years ago

This particular issue is on 0.6.4, probably not going to update it until I've gotten a way to migrate the configuration.

On 29 Jun 2017 12:39 pm, "Karsten Kraus" notifications@github.com wrote:

On the contrary. I'm really glad to have someone testing out all the little things that i've missed ;-) I'm just swamped with work right now so i am not sure when i'll get to look into it. Are you using the current 0.7-dev build or the older 0.6.4?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Argelbargel/gitlab-branch-source-plugin/issues/25#issuecomment-311929107, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgnvE_GloWQD5bAq_kGVPBrFRLrZv6Rks5sI37GgaJpZM4OJAWk .

iamfletch commented 7 years ago

First I want to say thanks to @Argelbargel for this amazing plugin. I am seeing the exact same issue here and I am running 0.6.4.1. @ace13 thanks for posting your work around snippet, saved me loads of time.

awilhelmer commented 7 years ago

I've got a (maybe stupid?) question about the workaround: Should I add these lines in every Jenkinsfile or its enough to have one empty project with a jenkinsfile containing these lines and I execute it when the listeners are lost?

ananace commented 7 years ago

My solution has been to write a meta project that runs and sets the listeners, though I guess you could stick it into an init.groovy.d script too. At least until the bug is fixed.

awilhelmer commented 7 years ago

Okay the listener readd project is working now. Thanks for that. But now all branches of a project are triggered instead of the branch where i pushed the commit.

awilhelmer commented 7 years ago

Found the problem. Two things are wrong:

  1. The ListenerInitializerTask is started without permissions. Starting the Thread with ACL.impersonate(ACL.SYSTEM, new ListenerInitializerTask()); resolves the problem.
  2. The ListenerInitializerTask is started to early. The projects arent loaded at this time. I fixed it with a sleep, not nice but working:
    public GitLabSCMWebHook() {
      manager = new HookManager();
      handler = new HookHandler();
      new Thread(new Runnable() {
         @Override
         public void run() {
            try {
               Thread.sleep(30000);
            }
            catch (InterruptedException e) {
               LOGGER.info("Thread Sleep FAILED!");
            }
            ACL.impersonate(ACL.SYSTEM, new ListenerInitializerTask());
         }
      }).start();
    }
Argelbargel commented 7 years ago

Ah. Ok. I think I've fixed some of it (ACL) in 0.7. But I'll have another look and fix it in 0.6.x too

Argelbargel commented 7 years ago

Jup. Was already fixed in 0.7-dev. Backported the changes to 0.6.4.2. Please give it a try and shout if it still does not work as expected...

ananace commented 7 years ago

The issue does seems to be fixed on 0.6.4.2, haven't needed to re-add any listeners even after restarting Jenkins multiple times.

Thanks.

Argelbargel commented 7 years ago

Ok. Then i'll close this. Thank you, guys!