jenkinsci / jms-messaging-plugin

https://plugins.jenkins.io/jms-messaging
12 stars 37 forks source link

Triggering builds randomly fails with NullPointerException #261

Open MichalHaluza opened 1 year ago

MichalHaluza commented 1 year ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.387.1 OS: Linux - 4.18.0-372.53.1.el8_6.x86_64 Java: 11.0.18 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- ace-editor:1.1 apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 authentication-tokens:1.4 authorize-project:1.4.0 bootstrap4-api:4.6.0-5 bootstrap5-api:5.2.1-3 bouncycastle-api:2.27 branch-api:2.1071.v1a_188a_562481 caffeine-api:3.1.6-115.vb_8b_b_328e59d8 checks-api:1.8.1 cloud-stats:267.v577e3742c282 cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8 cloudbees-folder:6.815.v0dd5a_cb_40e0e command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-36.vc008c8fcda_7b_ config-file-provider:3.11.1 configuration-as-code:1625.v27444588cc3d credentials:1224.vc23ca_a_9a_2cb_0 credentials-binding:604.vb_64480b_c56ca_ data-tables-api:1.12.1-4 dependency-check-jenkins-plugin:5.4.0 display-url-api:2.3.7 durable-task:504.vb10d1ae5ba2f echarts-api:5.4.0-1 font-awesome-api:6.2.1-1 git:5.0.1 git-client:4.2.0 git-server:99.va_0826a_b_cdfa_d gitlab-plugin:1.7.12 handlebars:3.0.8 htmlpublisher:1.31 instance-identity:142.v04572ca_5b_265 ionicons-api:45.vf54fca_5d2154 jackson2-api:2.15.0-334.v317a_165f9b_7c jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:226.v71211feb_e7e9 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-8 jaxb:2.3.8-1 jdk-tool:63.v62d2fd4b_4793 jersey2-api:2.39.1-1 jms-messaging:1.1.27 job-dsl:1.82 jquery3-api:3.6.1-2 jsch:0.1.55.61.va_e9ee26616e7 junit:1198.ve38db_d1b_c975 kerberos-sso:1.11 kubernetes:3896.v19b_160fd9589 kubernetes-client-api:6.4.1-215.v2ed17097a_8e9 kubernetes-credentials:0.10.0 ldap:671.v2a_9192a_7419d lockable-resources:1131.vb_7c3d377e723 mailer:448.v5b_97805e3767 matrix-auth:3.1.6 matrix-project:789.v57a_725b_63c79 metrics:4.2.13-420.vea_2f17932dd6 mina-sshd-api-common:2.9.2-62.v199162f0a_2f8 mina-sshd-api-core:2.9.2-62.v199162f0a_2f8 momentjs:1.1.1 okhttp-api:4.10.0-132.v7a_7b_91cef39c openstack-cloud:2.63 parameterized-scheduler:1.2 parameterized-trigger:2.45 pipeline-build-step:487.va_823138eee8b_ pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:629.vb_5627b_ee2104 pipeline-input-step:466.v6d0a_5df34f81 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2121.vd87fb_6536d1e pipeline-model-definition:2.2121.vd87fb_6536d1e pipeline-model-extensions:2.2121.vd87fb_6536d1e pipeline-rest-api:2.31 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2121.vd87fb_6536d1e pipeline-stage-view:2.31 pipeline-utility-steps:2.15.1 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:2.20.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 prometheus:2.1.1 rebuild:1.34 redhat-ci-plugin:1.5.16 resource-disposer:0.21 role-strategy:587.588.v850a_20a_30162 saml:4.385.v4dea_91565e9d scm-api:667.v8b_6e07cdc7f2 script-security:1244.ve463715a_f89c snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 sonar:2.15 splunk-devops:1.10.1 splunk-devops-extend:1.10.1 ssh-credentials:305.v8f4381501156 ssh-slaves:2.877.v365f5eb_a_b_eec sshd:3.275.v9e17c10f2571 structs:324.va_f5d6774f3a_d swarm:3.39 token-macro:359.vb_cde11682e0c trilead-api:2.84.v72119de229b_7 update-sites-manager:2.0.0 variant:59.vf075fe829ccb workflow-aggregator:596.v8c21c963d92d workflow-api:1208.v0cc7c6e0da_9e workflow-basic-steps:1010.vf7a_b_98e847c1 workflow-cps:3641.vf58904a_b_b_5d8 workflow-durable-task-step:1234.v019404b_3832a workflow-job:1292.v27d8cc3e2602 workflow-multibranch:733.v109046189126 workflow-scm-step:408.v7d5b_135a_b_d49 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.44 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

RHEL 8 (CSB supported instance of Jenkins)

Reproduction steps

Relevant JCasC configuration:

  jmsProviders:
    configs:
    - activeMq:
        name: "Red Hat UMB"
        broker: "ssl://umb.api.redhat.com:61616"
        authenticationMethod:
          sslCertificate:
            keystore: ***
            keypwd: ***
            truststore: ***
            trustpwd: ***
        topicProvider: "umb"
        useQueues: true

Relevant pipeline trigger:

    triggers {
        ciBuildTrigger {
            providers {
                providerDataEnvelope {
                    providerData {
                        activeMQSubscriber {
                            name("Red Hat UMB")
                            overrides {
                                topic("Consumer.rheldst-jenkins-csb.jenkins-errata-dispatcher.VirtualTopic.eng.errata.activity.status")
                            }
                            selector("errata_status='SHIPPED_LIVE' AND product='RHEL'")
                            variable("CI_MESSAGE")
                            timeout(60)
                        }
                    }
                }
            }
        }
    }

Expected Results

Each consumed message results in a successfully triggered / scheduled job.

Actual Results

Triggering jobs occasionally fails with

java.lang.NullPointerException
    at com.redhat.jenkins.plugins.ci.CIShouldScheduleQueueAction.shouldSchedule(CIShouldScheduleQueueAction.java:33)
    at hudson.model.Queue.scheduleInternal(Queue.java:606)
    at hudson.model.Queue.schedule2(Queue.java:576)
    at jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:158)
    at jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:123)
    at com.redhat.jenkins.plugins.ci.CIBuildTrigger.scheduleBuild(CIBuildTrigger.java:544)
    at com.redhat.jenkins.plugins.ci.messaging.JMSMessagingWorker.trigger(JMSMessagingWorker.java:76)
    at com.redhat.jenkins.plugins.ci.messaging.ActiveMqMessagingWorker.process(ActiveMqMessagingWorker.java:280)
    at com.redhat.jenkins.plugins.ci.messaging.ActiveMqMessagingWorker.receive(ActiveMqMessagingWorker.java:311)
    at com.redhat.jenkins.plugins.ci.threads.CITriggerThread.run(CITriggerThread.java:90) 

e.g. on 2023-06-13, our instance received 9 messages, but only 7 jobs were triggered. The other 2 failed with the above stack trace.

Looking into company-wide Splunk logs between 2023-05-01 and 2023-06-15, this has happened

The total number of jobs which have been triggered properly is currently unknown to me, but it is high enough so the instances appear to work almost normally. Each instance is a separate Jenkins master owned and configured by a different team.

Anything else?

No response