search

jenkinsci / bitbucket-branch-source-plugin

Bitbucket Branch Source Plugin
https://plugins.jenkins.io/cloudbees-bitbucket-branch-source
MIT License
216 stars 353 forks source link

Changeset detection broke between 866.vdea_7dcd3008e and 874.v659a_b_70f5e69 #812

Open friesoft opened 7 months ago

friesoft commented 7 months ago

Jenkins and plugins versions report

Environment ```text Bitbucket Server: 8.9.9 Jenkins: 2.444 # also not working with 2.445 OS: Linux - 3.10.0-1160.105.1.el7.x86_64 Java: 17.0.10 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 authentication-tokens:1.53.v1c90fd9191a_b_ authorize-project:1.7.1 basic-branch-build-strategies:81.v05e333931c7d bitbucket:241.v6d24a_57f9359 bootstrap4-api:4.6.0-6 bootstrap5-api:5.3.2-3 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1148.vce12cfcdf090 build-timeout:1.32 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.0.2 cloudbees-bitbucket-branch-source:866.vdea_7dcd3008e # downgraded from 874.v659a_b_70f5e69 cloudbees-folder:6.921.vfb_b_224371fb_4 command-launcher:107.v773860566e2e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 credentials:1319.v7eb_51b_3a_c97b_ credentials-binding:657.v2b_19db_7d6e6d data-tables-api:1.13.8-2 display-url-api:2.200.vb_9327d658781 durable-task:550.v0930093c4b_a_6 echarts-api:5.4.3-2 email-ext:2.104 emailext-template:1.5 extended-choice-parameter:376.v2e02857547b_a_ font-awesome-api:6.5.1-2 git:5.2.1 git-client:4.6.0 git-server:114.v068a_c7cc2574 gson-api:2.10.1-15.v0d99f670e0a_7 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 instance-identity:185.v303dc7c645f9 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.16.1-373.ve709c6871598 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jjwt-api:0.11.5-77.v646c772fddb_0 jobConfigHistory:1229.v3039470161a_d joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ jquery:1.12.4-1 jquery3-api:3.7.1-1 jsch:0.2.16-86.v42e010d9484b_ json-path-api:2.9.0-33.v2527142f2e1d junit:1259.v65ffcef24a_88 lockable-resources:1240.v43673a_a_7944a_ mailer:463.vedf8358e006b_ matrix-auth:3.2.1 matrix-project:822.824.v14451b_c0fd42 mercurial:1260.vdfb_723cdcc81 mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_ mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_ okhttp-api:4.11.0-172.vda_da_1feeb_c6e pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-config-history:1.6 pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:704.vc58b_8890a_384 pipeline-input-step:491.vb_07d21da_1a_fb_ pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2175.v76a_fff0a_2618 pipeline-model-definition:2.2175.v76a_fff0a_2618 pipeline-model-extensions:2.2175.v76a_fff0a_2618 pipeline-rest-api:2.34 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2175.v76a_fff0a_2618 pipeline-stage-view:2.34 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.8.0 popper-api:1.16.1-3 popper2-api:2.11.6-4 rebuild:330.v645b_7df10e2a_ resource-disposer:0.23 reverse-proxy-auth-plugin:1.7.7 scm-api:683.vb_16722fb_b_80b_ script-security:1326.vdb_c154de8669 simple-theme-plugin:176.v39740c03a_a_f5 snakeyaml-api:2.2-111.vc6598e30cc65 ssh-credentials:308.ve4497b_ccd8f4 sshd:3.322.v159e91f6a_550 structs:337.v1b_04ea_4df7c8 timestamper:1.26 token-macro:400.v35420b_922dcb_ trilead-api:2.133.vfb_8a_7b_9c5dd1 variant:60.v7290fc0eb_b_cd workflow-aggregator:596.v8c21c963d92d workflow-api:1291.v51fd2a_625da_7 workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3867.v535458ce43fd workflow-durable-task-step:1331.vc8c2fed35334 workflow-job:1400.v7fd111b_ec82f workflow-multibranch:783.va_6eb_ef636fb_d workflow-scm-step:415.v434365564324 workflow-step-api:657.v03b_e8115821b_ workflow-support:865.v43e78cc44e0d ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Red Hat Enterprise Linux Server 7.9

Reproduction steps

We are using the following function to gather changed yaml files:

@NonCPS
def gatherChangedFiles() {
    def changeLogSets = currentBuild.changeSets
    def files = []
    for (int i = 0; i < changeLogSets.size(); i++) {
        def entries = changeLogSets[i].items
        for (int j = 0; j < entries.length; j++) {
            def entry = entries[j]
            def fileList = new ArrayList(entry.affectedFiles)
            for (int k = 0; k < fileList.size(); k++) {
                def file = fileList[k]
                if (!"delete".equals(file.editType.name) && file.path.endsWith(".yaml")) {
                    files.add(file.path)
                }
            }
        }
    }
    files = files.unique()
    return files
}

This can be reproduced by a simple git commit in the repo which is being built, modifying one of the yaml files (function filters for Files ending with ".yaml"). With the old version this triggered an entry in the array, using the new version the array stays empty.

Expected Results

Using 874.v659a_b_70f5e69 the returned files variable (from our function) is empty. We would expect to still get all changed files from the function mentioned above

Actual Results

After upgrading from 866.vdea_7dcd3008e to 874.v659a_b_70f5e69 changeset detection doesn't work anymore.

Anything else?

No response

Are you interested in contributing a fix?

No response

lifeofguenter commented 7 months ago

potentially @andrey-fomin if you have a look please?

kenden commented 7 months ago

We are also seeing something strange after upgrading this plugin from 866.vdea_7dcd3008e to 874.v659a_b_70f5e69. I'm adding here because it seems related. It you don't think so, let me know and I'll create another issue.

A multi-branch pipeline with "Discover pull requests from origin" used to show this at the beginning of the logs:

git fetch --tags --force --progress -- ssh://git@bitbucket.redactedserver.com/repo_example.git +refs/heads/*:refs/remotes/origin/* 
git fetch --tags --force --progress -- ssh://git@bitbucket.redactedserver.com/repo_example.git +refs/pull-requests/38/from:refs/remotes/origin/PR-38
git fetch --tags --force --progress -- ssh://git@bitbucket.redactedserver.com/repo_example.git +refs/heads/main:refs/remotes/upstream/main

and after the upgrade it just shows:

git fetch --tags --force --progress -- ssh://git@bitbucket.redactedserver.com/repo_example.git +refs/heads/branch_of_pr_38:refs/remotes/origin/branch_of_pr_38

We don't have local git information about pull requests, which causes issues in scripts we use.

This PR: https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/796/files seems to be the cause: there are related changes to file BitbucketGitSCMBuilder.java

image image

mtheiss commented 7 months ago

This change also leads to SonarQube being unable to capture the changes and always reporting 0 changed files.

SCM collecting changed files in the branch
Could not find ref 'master' in refs/heads, refs/remotes, refs/remotes/upstream or refs/remotes/origin
SCM collecting changed files in the branch (done) | time=3ms
andrey-fomin commented 7 months ago

I'll check

andrey-fomin commented 7 months ago

I can't reproduce this issue. For me change log is empty for the first build and contains some changes in next builds if branch is updated.

Do you use default strategy to compute change log agains previous build? Or do you use ChangelogToBranch extension? I can guess that new fetching implementation doesn't work properly with ChangelogToBranch extension. @friesoft @kenden

friesoft commented 7 months ago

We do nothing else beside of calling the mentioned function. We always got (and rely on) the change delta from last build

I've attached our job config in case this is of any help

config.xml ```xml Konfigurationen false H H/4 * * * 86400000 36c31674-7dc1-4af2-9447-677b5393c8a1 false .* NONE true -1 -1 false H H/4 * * * 86400000 false https:///bitbucket 36c31674-7dc1-4af2-9447-677b5393c8a1 PROJECT (config_dev|config_test) 1 1 1 Jenkinsfile true false ^.*$ false ```
sha-root commented 7 months ago

Hi.

I confirm a similar problem. I configured additional "ref specs" for git fetch - to use in "git diff ..." for generating of FILES_CHANGELIST in first PR build when changeset is empty. And it stoped work after 866.vdea_7dcd3008e. I downgraded plugin versions to 866.vdea_7dcd3008e and it has been working again.

Before updating to the latest versions:

`using GIT_SSH to set credentials

git fetch --no-tags --force --progress -- git@bitbucket.org:ORG/PROJECT.git +refs/heads/SOURCEBRANCH:refs/remotes/origin/PR-0001 +refs/heads/:refs/remotes/origin/ # timeout=10`

After plugin updating to 866 higher version, it removes fetching of additional ref specs:

`using GIT_SSH to set credentials

git fetch --no-tags --force --progress -- git@bitbucket.org:ORG/PROJECT.git +refs/heads/feature/SOURCEBRANCH:refs/remotes/origin/feature/SOURCEBRANCH # timeout=10`

And I get an error in my diff command

git diff --name-only origin/develop...origin/feature/SOURCEBRANCH fatal: ambiguous argument 'origin/develop...origin/feature/SOURCEBRANCH': unknown revision or path not in the working tree.

Additional Ref Specs are set through git-plugin behaviour "Specify ref specs"

Знімок екрана 2024-02-21 о 15 03 15 Знімок екрана 2024-02-21 о 15 02 31
kenden commented 7 months ago

@andrey-fomin We use the default strategy also.

See the config.xml of a job with the issue ```xml Build pipeline for project redacted job-name-redacted false true 5 -1 false H H/4 * * * 86400000 false name_redacted https://bitbucket.companyname.com service_account_credentials REDACTED repo_name_redacted 1 1 1 false false false service_account_ssh_key -1 10 -1 -1 Jenkinsfile ```
andrey-fomin commented 7 months ago

@sha-root I believe your problem should be solved by #815. Please can you check release 876.v857269a_5f439

@kenden I don't understand why pattern +refs/heads/*:refs/remotes/origin/* is used for cloning. I don't see RefSpecsSCMSourceTrait in your configuration.

The only thing I can assume that your builds are executed in different folders (for example, multiple worker nodes or parallel builds). If branch was overwritten with forced push then commit from previous build can be missing in the current build folder. Fetching all branches with +refs/heads/*:refs/remotes/origin/* can increase the probability that commit from the previous build is available in the current build folder. But it can't guarantee it. If commit from previous build is missing then change log will be empty and message First time build. Skipping changelog. should be in the output.

@kenden do you use force pushes? Please can you try again with 876.v857269a_5f439?

sha-root commented 7 months ago

@andrey-fomin
Thanks! 🚀 Release 876.v857269a_5f439 has fixed the problem with additional Ref Specs

kenden commented 7 months ago

@andrey-fomin Thanks. We updated to 876.v857269a_5f439. We still don't get: +refs/pull-requests/XX/from:refs/remotes/origin/PR-XX like we used to:

git fetch --tags --force --progress -- ssh://git@bitbucket.redactedserver.com/repo_example.git +refs/pull-requests/38/from:refs/remotes/origin/PR-38

We allow force pushed for pull requests. But we protect the base branch (main).

But made changed to our scripting to not depend on it. So this is not a problem for us anymore.