search

jenkinsci / gitlab-plugin

A Jenkins plugin for interfacing with GitLab
https://plugins.jenkins.io/gitlab-plugin/
GNU General Public License v2.0
1.44k stars 618 forks source link

Intermittent 400 Bad Request errors #1461

Open dg424 opened 1 year ago

dg424 commented 1 year ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.346.3 OS: Linux - 5.15.0-1026-aws Java: 11.0.16.1 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- ace-editor:1.1 allure-jenkins-plugin:2.30.2 analysis-model-api:10.16.1 anchore-container-scanner:1.0.24 ansicolor:1.0.2 ant:475.vf34069fef73c antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 artifact-manager-s3:670.v0558a_cb_c82c2 artifactory:3.17.4 audit-trail:3.11 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-global-configuration:100.v4619b_45d2dfc aws-java-sdk-ec2:1.12.287-357.vf82d85a_6eefd aws-java-sdk-minimal:1.12.287-357.vf82d85a_6eefd blackduck-detect:7.0.0 blueocean:1.25.7 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.7 blueocean-commons:1.25.7 blueocean-config:1.25.7 blueocean-core-js:1.25.7 blueocean-dashboard:1.25.7 blueocean-display-url:2.4.1 blueocean-events:1.25.7 blueocean-git-pipeline:1.25.7 blueocean-github-pipeline:1.25.7 blueocean-i18n:1.25.7 blueocean-jira:1.25.7 blueocean-jwt:1.25.7 blueocean-personalization:1.25.7 blueocean-pipeline-api-impl:1.25.7 blueocean-pipeline-editor:1.25.7 blueocean-pipeline-scm-api:1.25.7 blueocean-rest:1.25.7 blueocean-rest-impl:1.25.7 blueocean-web:1.25.7 bootstrap5-api:5.2.0-3 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 build-blocker-plugin:1.7.8 build-monitor-plugin:1.13+build.202205140447 build-user-vars-plugin:1.8 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.5 cloudbees-bitbucket-branch-source:791.vb_eea_a_476405b cloudbees-folder:6.740.ve4f4ffa_dea_54 clover:4.13.0 command-launcher:1.2 commons-lang3-api:3.12.0-36.vd97de6465d5b_ conditional-buildstep:1.4.2 config-file-provider:3.11.1 configuration-as-code:1512.vb_79d418d5fc8 configuration-as-code-groovy:1.1 copyartifact:1.47 cors-filter:1.1 credentials:1143.vb_e8b_b_ceee347 credentials-binding:523.vd859a_4b_122e6 cucumber-reports:5.7.3 dark-theme:245.vb_a_2b_b_010ea_96 dashboard-view:2.447.vda_124dd35f11 data-tables-api:1.12.1-3 description-setter:1.10 display-url-api:2.3.6 docker-commons:1.21 docker-workflow:521.v1a_a_dd2073b_2e dtkit-api:3.0.1 durable-task:500.v8927d9fd99d8 echarts-api:5.3.3-1 email-ext:2.91 embeddable-build-status:255.va_d2370ee8fde envinject:2.875.v9b_9e962da_a_ec envinject-api:1.199.v3ce31253ed13 extended-read-permission:3.2 extensible-choice-parameter:1.8.0 external-monitor-job:192.ve979ca_8b_3ccd favorite:2.4.1 file-parameters:205.vf6ce13b_e5dee font-awesome-api:6.1.2-1 forensics-api:1.15.1 gerrit-trigger:2.37.1215-NOKIA git:4.11.5 git-client:3.11.2 git-server:1.11 github:1.34.5 github-api:1.303-400.v35c2d8258028 github-branch-source:1694.vd46793a_c4a_57 gitlab-api:5.0.1-78.v47a_45b_9f78b_7 gitlab-branch-source:642.v9ed86b_b_54384 gitlab-plugin:1.5.35 gradle:1.39.4 groovy:442.v817e6d937d6c handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.30 http_request:1.16 ionicons-api:24.va_745a_b_a_97fc3 ivy:2.2 jackson2-api:2.13.3-285.vc03c0256d517 jacoco:3.3.2 jakarta-activation-api:2.0.1-1 jakarta-mail-api:2.0.1-1 javadoc:226.v71211feb_e7e9 javax-activation-api:1.2.0-4 javax-mail-api:1.6.2-5 jaxb:2.3.6-1 jdk-tool:55.v1b_32b_6ca_f9ca jenkins-design-language:1.25.7 jersey2-api:2.37-1 jira:3.8 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.81 job-restrictions:0.8 jobConfigHistory:1171.v04b_66d78555e jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.0-4 jsch:0.1.55.61.va_e9ee26616e7 junit:1119.1121.vc43d0fc45561 kubernetes:1.31.3 kubernetes-client-api:5.12.2-193.v26a_6078f65a_9 kubernetes-credentials:0.9.0 ldap:2.12 leastload:3.0.0 locale:180.v207501dff9b_a_ lockable-resources:2.16 mailer:435.v79ef3972b_5c7 mapdb-api:1.0.9-28.vf251ce40855d mask-passwords:3.3 matrix-project:772.v494f19991984 maven-plugin:3.19 mercurial:1251.va_b_121f184902 metrics:4.2.10-389.v93143621b_050 mina-sshd-api-common:2.8.0-36.v8e25ce90d4b_1 mina-sshd-api-core:2.8.0-36.v8e25ce90d4b_1 momentjs:1.1.1 multibranch-build-strategy-extension:1.0.10 next-build-number:1.8 okhttp-api:4.9.3-108.v0feda04578cf overops-query:3.0.3 pam-auth:1.10 parameter-separator:1.3 parameterized-scheduler:1.1 parameterized-trigger:2.45 performance:3.20 pipeline-build-step:2.18 pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:612.v84da_9c54906d pipeline-input-step:449.v77f0e8b_845c4 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2114.v2654ca_721309 pipeline-model-definition:2.2114.v2654ca_721309 pipeline-model-extensions:2.2114.v2654ca_721309 pipeline-rest-api:2.24 pipeline-stage-step:296.v5f6908f017a_5 pipeline-stage-tags-metadata:2.2114.v2654ca_721309 pipeline-stage-view:2.24 pipeline-utility-steps:2.13.0 plain-credentials:139.ved2b_9cf7587b plugin-util-api:2.17.0 popper2-api:2.11.6-1 postbuildscript:3.1.0-375.v3db_cd92485e1 prism-api:1.28.0-2 prometheus:2.0.11 pubsub-light:1.16 rebuild:1.34 resource-disposer:0.20 robot:3.2.0 role-strategy:561.v9846c7351a_41 run-condition:1.5 saml:2.333.vc81e525974a_c scm-api:621.vda_a_b_055e58f7 script-security:1175.v4b_d517d6db_f0 sidebar-link:2.2.0 simple-theme-plugin:103.va_161d09c38c7 snakeyaml-api:1.31-84.ve43da_fb_49d0b sonar:2.14 sse-gateway:1.25 ssh-agent:295.v9ca_a_1c7cc3a_a_ ssh-credentials:305.v8f4381501156 sshd:3.242.va_db_9da_b_26a_c3 structs:324.va_f5d6774f3a_d subversion:2.16.0 support-core:1206.v14049fa_b_d860 synopsys-coverity:3.0.1 test-results-analyzer:0.3.5 theme-manager:1.5 throttle-concurrents:2.9 timestamper:1.19 token-macro:308.v4f2b_ed62b_b_16 trilead-api:1.67.vc3938a_35172f uno-choice:2.6.4 urltrigger:1.02 validating-string-parameter:2.8 variant:59.vf075fe829ccb view-job-filters:2.3 warnings-ng:9.19.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1192.v2d0deb_19d212 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:2759.v87459c4eea_ca_ workflow-durable-task-step:1199.v02b_9244f8064 workflow-job:1207.ve6191ff089f8 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:838.va_3a_087b_4055b ws-cleanup:0.43 xray-connector:2.6.1 xtrigger-api:0.4 xunit:3.1.0 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Ubuntu 22.04.1 LTS for both

Reproduction steps

The call that triggers the error is updateGitlabCommitStatus and the status is verified to be one of https://github.com/jenkinsci/gitlab-plugin/blob/master/src/main/java/com/dabsquared/gitlabjenkins/gitlab/api/model/BuildState.java

As mentioned in the title, this is an "intermittent" problem, so reproduction is pretty much hit or miss. The reason for raising this issue is to hopefully get some ideas about why it might be happening and how to workaround or even get a solution - eg. do we need a later version of the plugin ?

Expected Results

updateGitlabCommitStatus works all of the time.

Actual Results

2023-04-27 21:13:18.035+0000 [id=67297737] SEVERE c.d.g.util.CommitStatusUpdater#updateCommitStatus: Failed to update Gitlab commit status for project '2392' javax.ws.rs.BadRequestException: HTTP 400 Bad Request at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.handleErrorStatus(ClientInvocation.java:241) at org.jboss.resteasy.client.jaxrs.internal.proxy.extractors.DefaultEntityExtractorFactory$3.extractEntity(DefaultEntityExtractorFactory.java:50) at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:154) at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:115) at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76) at com.dabsquared.gitlabjenkins.gitlab.api.impl.$Proxy209.changeBuildStatus(Unknown Source) at com.dabsquared.gitlabjenkins.gitlab.api.impl.ResteasyGitLabClient.changeBuildStatus(ResteasyGitLabClient.java:110) at com.dabsquared.gitlabjenkins.gitlab.api.impl.AutodetectingGitLabClient$13.execute(AutodetectingGitLabClient.java:177) at com.dabsquared.gitlabjenkins.gitlab.api.impl.AutodetectingGitLabClient$13.execute(AutodetectingGitLabClient.java:174) at com.dabsquared.gitlabjenkins.gitlab.api.impl.AutodetectingGitLabClient$GitLabOperation.execute(AutodetectingGitLabClient.java:397) at com.dabsquared.gitlabjenkins.gitlab.api.impl.AutodetectingGitLabClient$GitLabOperation.access$100(AutodetectingGitLabClient.java:394) at com.dabsquared.gitlabjenkins.gitlab.api.impl.AutodetectingGitLabClient.execute(AutodetectingGitLabClient.java:391) at com.dabsquared.gitlabjenkins.gitlab.api.impl.AutodetectingGitLabClient.changeBuildStatus(AutodetectingGitLabClient.java:173) at com.dabsquared.gitlabjenkins.util.CommitStatusUpdater.updateCommitStatus(CommitStatusUpdater.java:85) at com.dabsquared.gitlabjenkins.util.CommitStatusUpdater.updateCommitStatus(CommitStatusUpdater.java:97) at com.dabsquared.gitlabjenkins.workflow.GitLabBuildsStep$GitLabBuildStepExecution$1.onStart(GitLabBuildsStep.java:87) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.launch(CpsBodyExecution.java:130) at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker.launch(CpsBodyInvoker.java:188) at org.jenkinsci.plugins.workflow.cps.DSL$ThreadTaskImpl.lambda$invokeBodiesAndSwitchToAsyncMode$0(DSL.java:784) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.withBodyInvokers(CpsStepContext.java:538) at org.jenkinsci.plugins.workflow.cps.DSL$ThreadTaskImpl.invokeBodiesAndSwitchToAsyncMode(DSL.java:774) at org.jenkinsci.plugins.workflow.cps.DSL$ThreadTaskImpl.eval(DSL.java:739) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:198) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:95) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)

Anything else?

No response

dg424 commented 1 year ago

Update: We have narrowed the issue to calling updateGitlabCommitStatus with the "same" state again or setting the state to a previous state (eg. from running to pending) . Is doing this invalid - i.e. cannot call with the same state multiple times or call setting the state back to a previous state ? Is this documented somewhere ?

krisstern commented 1 year ago

@dg424 I think for now the best documentation is the code itself unfortunately, but there is incentive to improve our documentation this summer as a relevant Google Summer of Code (GSoC) 2023 project is under way. According to your reasoning the problem could be in how this GitLabBuildsStep.java file is written. So this might indeed be a faulty business logic issue. I will have more time to look into it later. However, if you have found a way to patch this bug meanwhile, by all means a pull request would be more than welcome!