search

jenkinsci / bitbucket-branch-source-plugin

Bitbucket Branch Source Plugin
https://plugins.jenkins.io/cloudbees-bitbucket-branch-source
MIT License
217 stars 349 forks source link

Invalid credentials on Clone #816

Closed giri-vsr closed 3 months ago

giri-vsr commented 4 months ago

Jenkins and plugins versions report

Environment Jenkins: 2.446 OS: Linux - 5.4.17-2136.300.7.el8uek.aarch64 Java: 11.0.20 - Red Hat, Inc. (OpenJDK 64-Bit Server VM) --- ansicolor:1.0.4 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 authentication-tokens:1.53.v1c90fd9191a_b_ bitbucket-oauth:0.13 blueocean-autofavorite:1.2.5 blueocean-commons:1.27.11 blueocean-config:1.27.11 blueocean-core-js:1.27.11 blueocean-dashboard:1.27.11 blueocean-display-url:2.4.2 blueocean-events:1.27.11 blueocean-git-pipeline:1.27.11 blueocean-github-pipeline:1.27.11 blueocean-i18n:1.27.11 blueocean-jwt:1.27.11 blueocean-personalization:1.27.11 blueocean-pipeline-api-impl:1.27.11 blueocean-pipeline-editor:1.27.11 blueocean-pipeline-scm-api:1.27.11 blueocean-rest:1.27.11 blueocean-rest-impl:1.27.11 blueocean-web:1.27.11 bootstrap5-api:5.3.2-4 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1152.v6f101e97dd77 build-timeout:1.32 build-user-vars-plugin:1.9 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.0.2 cloudbees-bitbucket-branch-source:876.v857269a_5f439 cloudbees-folder:6.921.vfb_b_224371fb_4 command-launcher:107.v773860566e2e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 credentials:1319.v7eb_51b_3a_c97b_ credentials-binding:657.v2b_19db_7d6e6d data-tables-api:1.13.8-4 display-url-api:2.200.vb_9327d658781 durable-task:550.v0930093c4b_a_6 echarts-api:5.4.3-4 email-ext:2.104 favorite:2.208.v91d65b_7792a_c font-awesome-api:6.5.1-3 git:5.2.1 git-client:4.6.0 git-server:114.v068a_c7cc2574 github:1.38.0 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1772.va_69eda_d018d4 gradle:2.10 gson-api:2.10.1-15.v0d99f670e0a_7 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 hidden-parameter:237.v4b_df26c7a_f0e htmlpublisher:1.32 instance-identity:185.v303dc7c645f9 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.16.1-373.ve709c6871598 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.11 jersey2-api:2.41-133.va_03323b_a_1396 jjwt-api:0.11.5-77.v646c772fddb_0 joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ jquery3-api:3.7.1-2 jsch:0.2.16-86.v42e010d9484b_ json-path-api:2.9.0-33.v2527142f2e1d junit:1259.v65ffcef24a_88 lockable-resources:1243.v346d600eea_24 mailer:463.vedf8358e006b_ matrix-auth:3.2.1 matrix-project:822.824.v14451b_c0fd42 mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_ mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_ okhttp-api:4.11.0-172.vda_da_1feeb_c6e oracle-cloud-infrastructure-compute:1.0.17 pam-auth:1.10 pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:704.vc58b_8890a_384 pipeline-input-step:491.vb_07d21da_1a_fb_ pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2175.v76a_fff0a_2618 pipeline-model-definition:2.2175.v76a_fff0a_2618 pipeline-model-extensions:2.2175.v76a_fff0a_2618 pipeline-rest-api:2.34 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2175.v76a_fff0a_2618 pipeline-stage-view:2.34 pipeline-utility-steps:2.16.2 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:4.1.0 pubsub-light:1.18 resource-disposer:0.23 scm-api:683.vb_16722fb_b_80b_ script-security:1326.vdb_c154de8669 slack:684.v833089650554 snakeyaml-api:2.2-111.vc6598e30cc65 sse-gateway:1.26 ssh-credentials:308.ve4497b_ccd8f4 ssh-slaves:2.948.vb_8050d697fec sshd:3.322.v159e91f6a_550 structs:337.v1b_04ea_4df7c8 timestamper:1.26 token-macro:400.v35420b_922dcb_ trilead-api:2.133.vfb_8a_7b_9c5dd1 variant:60.v7290fc0eb_b_cd workflow-aggregator:596.v8c21c963d92d workflow-api:1291.v51fd2a_625da_7 workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3867.v535458ce43fd workflow-durable-task-step:1331.vc8c2fed35334 workflow-job:1400.v7fd111b_ec82f workflow-multibranch:783.va_6eb_ef636fb_d workflow-scm-step:415.v434365564324 workflow-step-api:657.v03b_e8115821b_ workflow-support:865.v43e78cc44e0d ws-cleanup:0.45

What Operating System are you using (both controller, and any agents involved in the problem)?

Oracle linux

Reproduction steps

Create a job with cron

when the cron run it fails to clone with invalid credentials

`#!/bin/groovy pipeline {

agent {
    node {
        label 'build'
    }
}
options {
    disableConcurrentBuilds()
}
triggers {
    cron('0 9 * * *')
}

stages {

    stage('CleanUp Container Images') {
        steps {
            sh "sudo make run"
        }
    }

}`

Expected Results

Clone happens successfully

Actual Results

Fails to clone when job is triggered by cron but works after Jenkins restart and triggered manually

`Fetching upstream changes from https://x-token-auth@bitbucket.org/xxxxxxx/xxxxxxx.git

git --version # timeout=10 git --version # 'git version 2.31.1' git fetch --no-tags --force --progress -- https://x-token-auth:{xxxxxxx}@bitbucket.org/xxxxxxx/xxxxxxx.git +refs/heads/master:refs/remotes/origin/master # timeout=10 ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Command "git fetch --no-tags --force --progress -- https://x-token-auth:{xxxxxxx}@bitbucket.org/xxxxxxx/xxxxxxx.git +refs/heads/master:refs/remotes/origin/master" returned status code 128: stdout: stderr: remote: Invalid credentials fatal: Authentication failed for 'https://bitbucket.org/xxxxxxx/xxxxxxx.git/'

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2842)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:871)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)`

Anything else?

No response

Are you interested in contributing a fix?

No response

cshabee commented 4 months ago

We experience the same issue very frequently for many of our "normal" pipelines. The last version that properly worked was the 866 one. The only way we can run those jobs is to trigger a manual pipeline scan and then it works for some time for that specific multibranch pipeline.

ManneW commented 4 months ago

Much like @cshabee, we've started to experience this issue recently and it is not limited to runs triggered by cron. The workaround we found was to save the configuration for the multibranch pipeline project (without doing any changes), which in turn triggers a pipeline scan. So next time we encounter the problem, we will try with @cshabee's suggestion and only trigger the scan :pray:

I can file a separate issue for the "general issue" (not tied to cron as the trigger) or do you think this issue can be used for tracking both cases, @giri-vsr ? :slightly_smiling_face:

giri-vsr commented 4 months ago

Yes you guys are right it is happening for normal pipeline as well. @cshabee Thanks for the scan work around. I am setting Periodically if not otherwise run value to 1 hour to see if it helps me avoid scan it manually.

@ManneW will update the title

cshabee commented 4 months ago

We have that option already set (to 1 day), it seems to not solve the issue, however setting it to a more frequent option might do the trick, I'm waiting for your findings, too.

thematrixdev commented 4 months ago

I see this issue on my Jenkins as well. I suspect Bitbucket has made some changes.

On 15th February (Hong Kong time), ssh bitbucket.org works, but git pull does not work. Same "auth failed". I have generated new SSH keys, added to bitbucket.org, tried again, failed. I have suspected if it is due to ED25519 key format. Tried RSA still did not work. Yet, using Gitkraken Bitbucket integration works.

On 16th, pulling with SSH still fails 99% of the time. One or two pull succeeded. Pulling on Jenkins failed.

On 21st, Bitbucket announced there was incident.

Some days later, whenever Jenkins fails to pull, I create a new app-password or oauth-token. And the issue persists till now.

I hope this provide some clues what Bitbucket might have changed.

giri-vsr commented 4 months ago

We have that option already set (to 1 day), it seems to not solve the issue, however setting it to a more frequent option might do the trick, I'm waiting for your findings, too.

Workaround setting it to 1 hour seems to be working.Haven't go any failure yet.

giri-vsr commented 3 months ago

Working fine after #824