Closed viceice closed 4 years ago
🤷♂ Feel free to revert until some kind soul contributes the code for Bitbucket 7
Ok, trying to build a fix.
Not seeing anything note worthy: https://developer.atlassian.com/server/bitbucket/reference/api-changelog/ so double 🤷♂
as a workaround i disable pr's and only build regular branches
Hmm sure your it is not a one time thing: https://docs.atlassian.com/bitbucket-server/rest/7.0.1/bitbucket-rest.html#idp280 this is still documented as is.
ok, updating to 7.01 to see if its a bitbucket bug
The at
field looks funky though 😓
yes, i think there should be the commit id.
You'd have to debug what the API is returning since that is somewhat unexpected 🤔
ok, np. i have some java experience. i also know the bitbucket api from our renovatebot 😅
Looks like 7.0.1 is working 🤔
😖
so closing this as it's working again. :man_shrugging:
There is something wrong.
git fetch +refs/pull-requests/575/from:refs/remotes/origin/PR-575
They should be equal. pr is opened from renovate/jint-3.0.x
to master
I'm also seeing this same issue with building PRs after upgrading to Bitbucket 7.0.1 yesterday. Anything I can do/test to help get this fixed?
Nope, disabled pr build for now. Currently the only fix is to downgrade.
stderr: fatal: Couldn't find remote ref refs/pull-requests/580/from
is gone with 7.0.1, but refs/pull-requests/580/from
is no longer updated by bitbucket after pushing new commits.
The git refrences pull request refs (refs/pull-requests/*) have never been supported by BitBucket / Atlassian as part of their API. https://community.atlassian.com/t5/Bitbucket-questions/Difference-of-refs-pull-requests-lt-ID-gt-merge-and-refs-pull/qaq-p/772142
OK, trying to send a pr which uses the rest api
Have we considered forcing the refs creation again by different call to API? Bad approach, but a quick-fix? (as discussed in https://issues.jenkins-ci.org/browse/JENKINS-61493)
I tried the latest RC https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/2.7.1-rc629.f242e60a14b4/ Doesn't appear to work properly. We have git-lfs on one of our repos and heavyweight checkout appears to be broken with git lfs. git lfs binary is installed on the jenkins master.
12:54:24 ERROR: Could not do lightweight checkout, falling back to heavyweight
12:54:24 java.io.FileNotFoundException: URL: /rest/api/1.0/projects/ML/repos/repo/browse/Jenkinsfile?at=pull-requests%2F219%2Fmerge&start=0&limit=500
12:54:24 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:831)
12:54:24 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getFileContent(BitbucketServerAPIClient.java:1123)
12:54:24 at com.cloudbees.jenkins.plugins.bitbucket.filesystem.BitbucketSCMFile.content(BitbucketSCMFile.java:98)
12:54:24 at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:335)
12:54:24 at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:107)
12:54:24 at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
12:54:24 at hudson.model.ResourceController.execute(ResourceController.java:97)
12:54:24 at hudson.model.Executor.run(Executor.java:428)
12:54:24 Checking out git https://bb.example.com/scm/ML/repo.git https://bb.example.com/scm/ML/repo.git into /var/jenkins_home/workspace/repo_PR-219@script to read Jenkinsfile
12:54:24 using credential jenkins
12:54:24 using credential jenkins
Which then later times out with:
12:54:40 Merging remotes/upstream/master commit a9bc1aac607fa24a97711736a18bfda3c651d1f1 into PR head commit 75dc89073f8627ecd7434f614138e726016d00a0
12:54:40 > git config core.sparsecheckout # timeout=10
12:54:40 > git checkout -f 75dc89073f8627ecd7434f614138e726016d00a0 # timeout=10
12:59:08 ERROR: Checkout failed
12:59:08 java.lang.InterruptedException
12:59:08 at java.lang.Object.wait(Native Method)
12:59:08 at java.lang.Object.wait(Object.java:502)
12:59:08 at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395)
12:59:08 at hudson.Proc$LocalProc.join(Proc.java:327)
12:59:08 at hudson.Proc.joinWithTimeout(Proc.java:172)
12:59:08 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2423)
12:59:08 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$1100(CliGitAPIImpl.java:81)
12:59:08 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2743)
12:59:08 at jenkins.plugins.git.MergeWithGitSCMExtension.checkout(MergeWithGitSCMExtension.java:144)
12:59:08 at jenkins.plugins.git.MergeWithGitSCMExtension.decorateRevisionToBuild(MergeWithGitSCMExtension.java:110)
12:59:08 at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1063)
12:59:08 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1168)
12:59:08 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
12:59:08 at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:155)
12:59:08 at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:142)
12:59:08 at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
12:59:08 at hudson.model.ResourceController.execute(ResourceController.java:97)
12:59:08 at hudson.model.Executor.run(Executor.java:428)
12:59:08 [Bitbucket] Notifying pull request build result
12:59:08 [Bitbucket] Build result notified
12:59:08 ERROR: Maximum checkout retry attempts reached, aborting
12:59:08 Finished: FAILURE
I can confirm this issue using Plugin Version 2.7.0 with Bitbucket Server 7.1.0
+1 on issue with BB 7.1.0 - our guys upgraded BB this weekend and all PRs no longer build (and we lean heavily on them) - is there a patch yet?
Both #294 and #302 is a workable solution.
Although #302 needs some refinements as it is basically a hack abusing the can merge api call. While it would be great to support both can merge and poking the PR changes api.
I back-ported #294 to a fork of 2.5.0 (I named it 2.5.1) because due to our setup we cannot upgrade to 2.6.0+ yet. So the question is - is there a process in Jenkins ecosystem to release backported versions for those who are stuck, like us? Obviously we can always build local versions, but would be nice to have official release somewhere...
My first question would be why are you stuck? But backporting is not uncommon but I would need a good reason.
Not sure it qualifies as 'good' reason, but we are CloudBees Enterprise customers, and their managed masters have a very specific plugin version set, usually 3-4 mos behind the times. For past few releases they have been fixed at 2.5.0 :-(. Often it will not be a big deal to update some plugins, their software will complain, but all will be fine, however when updating to 2.6.0+, there is a dependency hell that requires a who load of other plugins to be updated, and things start breaking left and right :-(. I went down that road a few times and it caused havoc, and I had to revert things (I really want that 2.6.0). Having 2.5.1, however, let me bring all our masters back in business quickly and relatively safely.
Honestly that's not a good reason for backporting, the dependencies which are updated should not cause dependency hell... if CloudBees is that far behind on updating plugins I am not sure what their managed masters are good for... 😅 2.6.0 was released on Nov 7, 2019 that's 5 months ago.
You are not entirely wrong, but there are other reasons for using their product that is not specific to versions of plugins... I do agree that 5 mos is way too long to be behind and expressed that to them :-)
Hi All, for what it's worth we've now hit this issue on our multiple Jenkins instances across our business (our central infra team upgraded to BB 7.0.1 over the weekend). Have stumbled across this issue in the process. As we've been debugging with our infra team, we've come across this: https://jira.atlassian.com/browse/BSERV-12284?focusedCommentId=2389584&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-2389584 Looks like it's in response to a comment where this thread is linked. Very keen to test an RC or release when it's available.
Hi again, can confirm that the following RC build: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/2.7.1-rc629.f242e60a14b4/ (link copied from above) solves the problem for us. We still see the "Lightweight error" at the beginning of the run, however it then continues without issue.
We got hit by the same problem after upgrade to Bitbucket Server 7.1.2. I installed the latest RC version from https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/2.7.1-rc632.56cea7d4b7e6/ but now the PR build fails on fetching the PR with the following error:
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --no-tags --progress -- https://idesuite.hlb.helaba.de/bitbucket/scm/MX/tool.build_deploy.git +refs/pull-requests/34/from:refs/remotes/origin/PR-34" returned status code 128:
stdout:
stderr: fatal: Couldn't find remote ref refs/pull-requests/34/from
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:798)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1122)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:155)
at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:142)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:428)
ERROR: Error cloning remote repo 'origin'
Does this need a minimum git version on the jenkins agent? We have the default RHEL 7 Version 1.8.3.1.
both Jenkins master and agent will need git to do heavy checkouts, even if the Jenkins master only needs the Jenkinsfile.
We have git, I was asking whether we need a version > 1.8.3.1. I was suspicious that the remote ref in the error message was incomplete (everything after the '/from' is missing).
We got hit by the same problem after upgrade to Bitbucket Server 7.1.2. I installed the latest RC version from https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/2.7.1-rc632.56cea7d4b7e6/ but now the PR build fails on fetching the PR with the following error:
ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Command "git fetch --no-tags --progress -- https://idesuite.hlb.helaba.de/bitbucket/scm/MX/tool.build_deploy.git +refs/pull-requests/34/from:refs/remotes/origin/PR-34" returned status code 128: stdout: stderr: fatal: Couldn't find remote ref refs/pull-requests/34/from at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:798) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1122) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:155) at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:142) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:428) ERROR: Error cloning remote repo 'origin'
Does this need a minimum git version on the jenkins agent? We have the default RHEL 7 Version 1.8.3.1.
From what I can tell there hasn't been any changes to the plugin that would require a change to the version of git you're using. All that has been changed is a call the the Bitbucket API. Are you sure you've installed the RC plugin correctly (you can check it on the 'installed' tab in the plugins section of Jenkins)? I only ask because the error you've quoted looks very similar to the one quoted in the first post above (after you read past the REST API failure)
Yes, the plugin seems to be installed correctly. I had to delete the PR including the commits and created them from scratch, then issued a new PR. Now the Jenkinsfile was not found by the Scan. After I opened the Diff tab of the PR (getting a Bitbucket error on the first try), the build was triggered. This seems to be another issue. I'll read up on it.
Ah yes, when our BB server was first upgraded we did see that we had to delete and recreate some PRs, since upgrading to the RC build of the plugin though we've had no issues. (I support multiple Jenkins servers across our estate, all pointing at the same BB server. Since installing the RC build the only problem we've had is the one I mentioned above about the lightweight checkout failing).
On Wed, 29 Apr 2020, 11:56 tomk3003, notifications@github.com wrote:
Yes, the plugin seems to be installed correctly. I had to delete the PR including the commits and created them from scratch, then issued a new PR. Now the Jenkinsfile was not found by the Scan. After I opened the Diff tab of the PR (getting a Bitbucket error on the first try), the build was triggered. This seems to be another issue. I'll read up on it.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jenkinsci/bitbucket-branch-source-plugin/issues/287#issuecomment-621127228, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACQ62ZBIWNIWVCMOR5O5BZDRPABUNANCNFSM4LEBNNJQ .
After reading a lot, I have a question about the current status:
With the current RC the PRs are only detected after manually opening the PR in Bitbucket and then manually triggering a pipeline scan? At least this is the status of our installation.
And this will perhaps be fixed with PR #302 ?
Please correct me if I'm wrong.
Before the upgrade, how were you triggering PR detection in Jenkins? We use the "Post Webhooks" plugin in Bitbucket to hit this plugin in Jenkins at the /scm-notify..... URL (sorry, can't remember the full URL and I'm replying from my phone as not at work today). Note that if you're now using the built-in Bitbucket "Webhooks" feature, you need to change the URL slightly to include the URL of the Bitbucket server when the request is sent. This isn't something new or different in the plugin (it's documented quite clearly already on the Jenkins plugin site for this plugin) it's just an idea for something to look at in the configuration of your repo in Bitbucket.
The Webhooks feature that's built into Bitbucket now has all sorts of features for debugging the Webhooks as well as determining when they're triggered.
This is all under the assumption you're using Webhooks to trigger the builds (makes sense if you're using this plug-in).
On Wed, 29 Apr 2020, 15:47 tomk3003, notifications@github.com wrote:
After reading a lot, I have a question about the current status:
With the current RC the PRs are only detected after manually opening the PR in Bitbucket and then manually triggereing a pipeline scan? At least this is the status of our installation.
And this will perhaps be fixed with PR #302 https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/302 ?
Please correct me if I'm wrong.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jenkinsci/bitbucket-branch-source-plugin/issues/287#issuecomment-621259614, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACQ62ZAFAW5LDA4CZTOFX4DRPA4YNANCNFSM4LEBNNJQ .
@tomk3003 in jenkins settings https://jenkins/configure make sure that
"Webhook implementation to use" is set to "native". Plugin webhook no longer appears to work with PR. Once you change that, re-scan the projects and you should see that jenkins set a webhook in BB in Webhooks (in project settings).
@dcherniv I'm sorry but I have to disagree with it not working with the "Plugin webhook" as it's working for us. HOWEVER, we've never used the "Manage hooks" feature in this plugin, we've always set the Webhooks manually in Bitbucket as we have a job to automate the creation of our Bitbucket repos with common settings. I can definitely confirm though that we have PRs being triggered, using the "Post Webhooks" Bitbucket plugin, to this SCM plugin, without any problem.
We were using native Webhooks from the start. The webhooks are working. For merging PRs everything works fine. But the PRs are not recognized after creation. The scan says "Checking PR-..., Jenkinsfile not found". If I visit the PR in bitbucket and do a rescan, the PR is recognized and the pipeline runs. I have checked the webhook status in Bitbucket: the events have fired and have a successful status.
So what happens when the webhook fires for PR, does Jenkins just not do anything? You have to manually scan it? Because you shouldn't have to manually scan it at all if you're using Webhooks.
On Wed, 29 Apr 2020, 16:18 tomk3003, notifications@github.com wrote:
We were using native Webhooks from the start. The webhooks are working. For merging PRs everything works fine. But the PRs are not recognized after creation. The scan says "Checking PR-..., Jenkinsfile not found". If I visit the PR in bitbucket and do a rescan, the PR is recognized and the pipeline runs. I have checked the webhook status in Bitbucket: the events have fired and have a successful status.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jenkinsci/bitbucket-branch-source-plugin/issues/287#issuecomment-621278433, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACQ62ZH2EDNDRXNKV33RWBDRPBAD3ANCNFSM4LEBNNJQ .
We use jenkins with native webhook mode and the plugin build from this pr. PR are visible and where build in jenkins. Only caveat it the heavy checkout.
Exactly, what happens is: Nothing. Even if I go to Jenkins and do a Pipeline Scan for the repo, nothing happens because the Jenkinsfile is not found. Then I go to Bitbucket, view the diff of the PR go back to Jenkins, do a Pipeline Scan again. And only then the Jenkinsfile is recognized and the pipeline is run. I think the problem has to be with getting the repo contents via the PR.
@viceice : I wouldnt mind the heavy checkout, we have activated "Wipe out repository & force clone" anyway for other reasons.
Perhaps this only happens when the PR is done from a fork. In this comment in #294 the same behaviour is mentioned.
For those who installed RC version and still see the problem - make sure you install the correct one https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/cloudbees-bitbucket-branch-source/2.7.1-rc629.f242e60a14b4/ We installed the newer RC version today and still see the problem. Installing exactly this one solved it for us - building pull requests on Jenkins work back again for us.
@wololock great spot, I would have just assumed that the latest RC was the same content as the one I posted a couple of days back.
@tomk3003 I'd recommend using the same version of the plugin mentioned by me and @wololock above and see if your issue resolves itself.
Your checklist for this issue
[x] Jenkins version: 2.204.5
[x] Plugin version: 2.7.0
[ ] Bitbucket cloud
[x] Bitbucket server and version: 7.0.0, 7.0.1
Description
This plugin is no longer able to fetch and build pull-requests. Looks like thay changed the api without writing a note.
https://community.atlassian.com/t5/Bitbucket-questions/Bitbucket-Server-7-0-PR-fetch-from-not-working/qaq-p/1320053