search

ashwanthkumar / gocd-build-github-pull-requests

GoCD plugin to build PRs for a Github repo
95 stars 72 forks source link

Pull Request Plugin Keeps Failing With Exit Error 128 #99

Open kenluberts opened 7 years ago

kenluberts commented 7 years ago

We keep seeing the same issue with the plugin to kick off builds with PR's. We have this occurring with multiple pipelines. Any help would be appreciated. This happens whether we use the https URL or the git ssh url for the same repo. Just a note: I have redacted our git repo info here.

2017-03-22 10:47:10,490 WARN [94@MessageListener for MaterialUpdateListener] GitHubPRBuildPlugin:67 - get latest revisions since: java.lang.RuntimeException: Exception Occurred: [git, clone, --branch=develop, https://github.com/xxxxx/xxxxx.com.git, /home/go-server/pipelines/flyweight/96ff1f4b-3f21-4bf2-9e41-1bf9decabf4a] - null at com.tw.go.plugin.cmd.Console.runOrBomb(Console.java:33) at com.tw.go.plugin.git.GitCmdHelper.runAndGetOutput(GitCmdHelper.java:397) at com.tw.go.plugin.git.GitCmdHelper.cloneRepository(GitCmdHelper.java:55) at com.tw.go.plugin.GitHelper.cloneOrFetch(GitHelper.java:38) at in.ashwanthkumar.gocd.github.GitHubPRBuildPlugin.handleLatestRevisionSince(GitHubPRBuildPlugin.java:249) at in.ashwanthkumar.gocd.github.GitHubPRBuildPlugin.handle(GitHubPRBuildPlugin.java:124) at com.thoughtworks.go.plugin.infra.DefaultPluginManager$2.execute(DefaultPluginManager.java:186) at com.thoughtworks.go.plugin.infra.DefaultPluginManager$2.execute(DefaultPluginManager.java:181) at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:315) at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245) at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:181) at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:32) at com.thoughtworks.go.plugin.access.scm.SCMExtension.latestModificationSince(SCMExtension.java:123) at com.thoughtworks.go.server.service.materials.PluggableSCMMaterialPoller.modificationsSince(PluggableSCMMaterialPoller.java:78) at com.thoughtworks.go.server.service.materials.PluggableSCMMaterialPoller.modificationsSince(PluggableSCMMaterialPoller.java:47) at com.thoughtworks.go.server.service.MaterialService.modificationsSince(MaterialService.java:109) at com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:53) at com.thoughtworks.go.server.materials.PluggableSCMMaterialUpdater.insertLatestOrNewModifications(PluggableSCMMaterialUpdater.java:61) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.insertLatestOrNewModifications(MaterialDatabaseUpdater.java:155) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterialWithNewRevisions(MaterialDatabaseUpdater.java:147) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$2.doInTransaction(MaterialDatabaseUpdater.java:110) at com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:24) at com.thoughtworks.go.server.transaction.TransactionTemplate$3.doInTransaction(TransactionTemplate.java:53) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:49) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:107) at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:50) at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:30) at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:69) at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:50) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 128 (Exit value: 128) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:153) at com.tw.go.plugin.cmd.Console.runOrBomb(Console.java:25) ... 30 more

ashwanthkumar commented 7 years ago

It looks like the git clone command is failing. Can you please help run this command manually on the server and share the output? 

$ git clone --branch=develop https://github.com/xxxxx/xxxxx.com.git /home/go-server/pipelines/flyweight/96ff1f4b-3f21-4bf2-9e41-1bf9decabf4a
kenluberts commented 7 years ago

I have changed this to use the ssh url... The git clone does work properly with that URL from the command line on the server.

Cloning into '/home/go-server/pipelines/flyweight/96ff1f4b-3f21-4bf2-9e41-1bf9decabf4a'... remote: Counting objects: 76558, done. remote: Compressing objects: 100% (134/134), done. remote: Total 76558 (delta 130), reused 65 (delta 65), pack-reused 76358 Receiving objects: 100% (76558/76558), 24.06 MiB | 39.00 KiB/s, done. Resolving deltas: 100% (52756/52756), done.

Here's a new exception with the ssh URL:

2017-03-23 13:24:55,443 INFO [92@MessageListener for MaterialUpdateListener] GitHubPRBuildPlugin:52 - Flyweight: /home/go-server/pipelines/flyweight/9efef14c-a62d-439e-be63-04dfdf729b50 2017-03-23 13:24:55,459 WARN [92@MessageListener for MaterialUpdateListener] GitHubPRBuildPlugin:67 - get latest revision: java.lang.RuntimeException: Exception Occurred: [git, clone, --branch=develop, git@github.com/xxxxx/xxxxx.com.git, /home/go-server/pipelines/flyweight/9efef14c-a62d-439e-be63-04dfdf729b50] - null at com.tw.go.plugin.cmd.Console.runOrBomb(Console.java:33) at com.tw.go.plugin.git.GitCmdHelper.runAndGetOutput(GitCmdHelper.java:397) at com.tw.go.plugin.git.GitCmdHelper.cloneRepository(GitCmdHelper.java:55) at com.tw.go.plugin.GitHelper.cloneOrFetch(GitHelper.java:38) at in.ashwanthkumar.gocd.github.GitHubPRBuildPlugin.handleGetLatestRevision(GitHubPRBuildPlugin.java:218) at in.ashwanthkumar.gocd.github.GitHubPRBuildPlugin.handle(GitHubPRBuildPlugin.java:122) at com.thoughtworks.go.plugin.infra.DefaultPluginManager$2.execute(DefaultPluginManager.java:186) at com.thoughtworks.go.plugin.infra.DefaultPluginManager$2.execute(DefaultPluginManager.java:181) at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:315) at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245) at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:181) at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:32) at com.thoughtworks.go.plugin.access.scm.SCMExtension.getLatestRevision(SCMExtension.java:109) at com.thoughtworks.go.server.service.materials.PluggableSCMMaterialPoller.latestModification(PluggableSCMMaterialPoller.java:62) at com.thoughtworks.go.server.service.materials.PluggableSCMMaterialPoller.latestModification(PluggableSCMMaterialPoller.java:47) at com.thoughtworks.go.server.service.MaterialService.latestModification(MaterialService.java:105) at com.thoughtworks.go.server.materials.LegacyMaterialChecker.findLatestModification(LegacyMaterialChecker.java:49) at com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:53) at com.thoughtworks.go.server.materials.ScmMaterialUpdater.addNewMaterialWithModifications(ScmMaterialUpdater.java:73) at com.thoughtworks.go.server.materials.PluggableSCMMaterialUpdater.addNewMaterialWithModifications(PluggableSCMMaterialUpdater.java:66) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.addNewMaterialWithModifications(MaterialDatabaseUpdater.java:177) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.initializeMaterialWithLatestRevision(MaterialDatabaseUpdater.java:134) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.access$000(MaterialDatabaseUpdater.java:47) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$1.doInTransaction(MaterialDatabaseUpdater.java:95) at com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:24) at com.thoughtworks.go.server.transaction.TransactionTemplate$3.doInTransaction(TransactionTemplate.java:53) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) at com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:49) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:92) at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:50) at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:30) at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:69) at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:50) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 128 (Exit value: 128) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:153) at com.tw.go.plugin.cmd.Console.runOrBomb(Console.java:25) ... 33 more

ashwanthkumar commented 7 years ago

The git command is failing with exit code of 128 when launched via the plugin. Not exactly sure why. Somethings I can think of could be

  1. Does /home/go-server/pipelines/flyweight/96ff1f4b-3f21-4bf2-9e41-1bf9decabf4a belong to go user? If you've manipulated some files inside it manually, git when launched by go user might not be able to remove / overwrite it.
  2. git command in PATH for the go user is probably wrong or points to an older version?

Also when you run the command manually, you ran it as a go user right? If not, can you please re-try that as a go user and see if that still works or if you happen to see any issues with it?

kenluberts commented 7 years ago

Hi - Sorry for the long delay in writing back...

Yes i ran everything as the go user to ensure that git does work from the box itself (and it does). Nothing in that flyweight directory should have been manipulated. Also, this issue affects one or two other pipelines that we have too...so it wouldn't be just this one flyweight directory.

Access to the go servers is kept to a minimum to prevent issues like that. On the version of the git client, we only have the one installed from what I can tell.

Would there be any additional debugging that I can enable for the plugin that might help us figure out what's going on?

rpm -qa | grep git- git-1.8.3.1-6.el7_2.1.x86_64

mcico commented 7 years ago

We face most probably the same issue. It used to work properly, but the suddenly stopped working. I deleted the offending clone directory and then it worked again for a few commits and stopped again with the same error.

We use gocd in version 17.3, the git in version 2.8.0 and the Git Feature Branch plugin in version 1.3.3, the problem already occurred with the plugin version 1.3.1.

This is the error I see in the gocd UI: Interaction with plugin with id 'git.fb' implementing 'scm' extension failed while requesting for 'latest-revisions-since'. Reason: [The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: '"Exception Occurred: [git, log, 301b0dda4fd9fff1342645fcc82829a0e29d2d72.., --date\u003diso, --pretty\u003dmedium] - /var/opt/go-server/pipelines/flyweight/4bdb4995-193c-4180-a9ea-518bd6bac09b"']

schnoddelbotz commented 6 years ago

I'm facing the same issue as @mcico -- the clone wasn't the issue for me, but the git log rev.. fails for me, as the named commit vanished due to a rebase. I think that situation should be handled, maybe by discarding information about last built PR/rev?

Anyhow, just removing the corresponding flyweight directory does not fix it at all -- a new clone will happen and then it tries to git log the non-existing rev again. I desperately tried to point to another commit id in cruise.h2.db (table modifications) -- it didn't help. @ashwanthkumar: Where is the plugin getting/setting the information about last built commit? Is there an easy way to reset it / the plugin?

ashwanthkumar commented 6 years ago

@schnoddelbotz We need the latest commit information to find if there are any new changes since the last time we checked the repository. Having said that, the GoCD's native Git material handles missing commits gracefully (in the recent versions) while this plugin doesn't. That's on my list for a while now. I just don't have enough time to work on it, if you would like to take a stab I would be happy to review.

On the modifications on the DB side, I'm not sure if that's even possible and I have not tried it either, sorry.

cheeseypi commented 6 years ago

I am getting a similar error

Interaction with plugin with id 'github.pr' implementing 'scm' extension failed while requesting for 'checkout'. Reason: [The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: '"Exception (Process exited with an error: 128 (Exit value: 128)) Occurred: [git, clone, --branch\u003dmaster, http://***[Valid GitHub Enterprise URL]***, /var/lib/go-agent/pipelines/image-pr-testing] - null"']

Running the clone command manually from the same server results in a success, but only if I replace \u003d with =. This issue began happening suddenly for no obvious reason -- it simply began failing even though no changes were made to the pipeline from the last success.

Using GoCD 17.4.0, plugin version 1.3.4

EDIT: I have had success manually cleaning the .cache and .git files from /var/lib/go-agent/pipelines/image-pr-testing every few runs.

dokaspar commented 6 years ago

Totally agree with @cheeseypi. Every once in a while, I get the same error with a suspicious --branch\u003dmaster in the log output.

Also the observations of @mcico with --date\u003diso, --pretty\u003dmedium in the log statement indicates that this is some kind of encoding issue.

drawm commented 6 years ago

Same problem here with Bitbucket Server...

Interaction with plugin with id 'stash.pr' implementing 'scm' extension failed while requesting for 'checkout'. Reason: [The plugin sent a response that could not be understood by Go. 
Plugin returned with code '500' and the following response: 
'"Exception (Process exited with an error: 128 (Exit value: 128)) Occurred: [git, clone, --branch\u003dmaster, ssh://git@***************************:****/***/*********.git, /go/pipelines/************/************] - null"']
ashwanthkumar commented 6 years ago

Generally the above set of errors can be resolved by deleting the checkout directory. While the general hypothesis is that the flyweight directory is corrupted, I have still not been able to figure out why / what is causing it to be that way. Could be a network issue or something else as well. I would be happy to get on call via Skype / Hangouts with someone who is interested to help debug an offending flyweight directory remotely.

peterwillis commented 6 years ago

It would be really handy if the complete command output was shown in the error, instead of just the exit value. Is it in a log file somewhere?

ashwanthkumar commented 6 years ago

@peterwillis We don't log the command. The main reason being sometimes folks add the materials with credentials as part of the URI. If you would like to give it a stab at the change, you can find the relevant code here. I would be happy to review and merge the changes.

peterwillis commented 6 years ago

Ok thank you for the response.

linnzha commented 5 years ago

Hi, i have a similar issue:

su go

Go Version: 18.6.0 plugin version:1.4.0-RC2 git version 2.15.2

when use go user to manually execute git clone is ok: git clone --branch=develop https://github.opco.com/user/xx.git /go-working-dir/pipelines/flyweight/0db6623a-90a3-4d6a-9a87-1e6bb87e5dfb input username input token succ,,,

Many thank you if who can tell me what happens...

ashwanthkumar commented 5 years ago

@linnzha Just a few questions, you seem to be using the HTTPS url. Are username and password added as part of the material configuration? Also does using the SSH URL help solve this problem? (Assuming you've ssh keys setup already)

linnzha commented 5 years ago

Thank you for reply.

But when config Username and Password(no matter password or token), and click button for check connection will failed.

fail message is: check connection failed. Server returned HTTP response code: 401 for URL: https://api.github.com/repos/user/xx

I was doubt why message still show api.github.com error because I have config new endpoint in .github...

And when I googles github ssh url, found it could not use ssh url... "If you are accessing an organization that uses SAML single sign-on, you won't be able to clone with SSH. Instead, clone with the HTTPS URL."

@linnzha Just a few questions, you seem to be using the HTTPS url. Are username and password added as part of the material configuration? Also does using the SSH URL help solve this problem? (Assuming you've ssh keys setup already)

ashwanthkumar commented 5 years ago

@linnzha Can you try the following and let me know

linnzha commented 5 years ago

So many thank you for your reply. @ashwanthkumar

Has solved the issue. Directly config the material url as https://{token}@github.opco.net/user/xx.git

dawncold commented 5 years ago

@ashwanthkumar I faced the same issue like @linnzha, I used gocd server and agent docker image.

run gocd-server: 

docker run -d -v /path/to/godata:/godata -v /path/to/gohome:/home/go -p8153:8153 -p8154:8154 gocd/gocd-server:v19.2.0

run gocd-agent:

docker run -d -e GO_SERVER_URL=https://172.17.0.1:8154/go gocd/gocd-agent-alpine-3.8:v19.2.0

and I created .github file under /path/to/gohome, the content:

login=<github enterprise username>
password=<github enterprise personal token>
endpoint=https://github.enterprise.com/api/v3

When I check connection during add Github material(only filled url and leave user, password blank), it returns: check connection failed. Server returned HTTP response code: 401 for URL: https://api.github.com/xxx

it should use my endpoint, but it used default github API endpoint.

and after start pipeline,

Material sourcecode checkout failed: Interaction with plugin with id 'github.pr' implementing 'scm' extension failed while requesting for 'checkout'. Reason: [The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: '"Exception (Process exited with an error: 128 (Exit value: 128)) Occurred: [git, clone, --branch\u003dmaster, https://github.enterprise.com/xxx/xxx.git, /go/pipelines/pr/app1] - null"']
linnzha commented 5 years ago

@ashwanthkumar I faced the same issue like @linnzha, I used gocd server and agent docker image.

run gocd-server:

docker run -d -v /path/to/godata:/godata -v /path/to/gohome:/home/go -p8153:8153 -p8154:8154 gocd/gocd-server:v19.2.0

run gocd-agent:

docker run -d -e GO_SERVER_URL=https://172.17.0.1:8154/go gocd/gocd-agent-alpine-3.8:v19.2.0

and I created .github file under /path/to/gohome, the content:

login=<github enterprise username>
password=<github enterprise personal token>
endpoint=https://github.enterprise.com/api/v3

When I check connection during add Github material(only filled url and leave user, password blank), it returns: check connection failed. Server returned HTTP response code: 401 for URL: https://api.github.com/xxx

it should use my endpoint, but it used default github API endpoint.

and after start pipeline,

Material sourcecode checkout failed: Interaction with plugin with id 'github.pr' implementing 'scm' extension failed while requesting for 'checkout'. Reason: [The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: '"Exception (Process exited with an error: 128 (Exit value: 128)) Occurred: [git, clone, --branch\u003dmaster, https://github.enterprise.com/xxx/xxx.git, /go/pipelines/pr/app1] - null"']

Did you try to config the material url like below? https://{token}@github.opco.net/user/xx.git

dawncold commented 5 years ago

@linnzha no, I just use the URL above and I have read the plugin source code, it should use my endpoint, and username, but it doesn't work as my think

ashwanthkumar commented 5 years ago

@dawncold If you're using the Oauth Token as the password, can you change the password as oauth in your ~/.github file?

linnzha commented 5 years ago

@linnzha no, I just use the URL above and I have read the plugin source code, it should use my endpoint, and username, but it doesn't work as my think

previously i was also faced with issue for pr plugin use common api endpoint, not use company api endpoint, it seems so strange because i think i have config opco api in .github file, but when change material url to use token url, it is ok...

parasquid commented 2 years ago

hello! just encountered this issue as well

"Exception (Process exited with an error: 128 (Exit value: 128)) Occurred: [git, clone, --branch\u003dmaster, https://github.com/xxx/yyy.git, /go-working-dir/pipelines/flyweight/5f7f7c9c-f371-404f-a5e2-47fe1773d642] - null"

and just like what was mentioned here, adding the token in the https endpoint fixes things.

I should note that this only seems to happen with private repos; pipelines using public repos work properly.

I've setup ~/.github in the server's as well (check connection works with the https without the token, and the ssh url) with the token as oauth key's value:

login=parasquid
oauth=ghp_yyy

While this is fine in the meantime, there will be some issues further on with the token being naked as part of the url. Would love to have the token only present in the home folder of the server, as that can be more easily secured than config pages.

deesweet commented 1 year ago

Hello - noting that I've observed this issue also. It popped up while testing an upgrade from GoCD server 22.2.0 to 23.1.0 using Docker. I've tried all of the workarounds mentioned in this issue thread, including adding the token to the SCM material HTTPS URL, and changing the URL from HTTPS to SSH, and the issue seems to persist despite a multiple resets of the GoCD server Docker test environment (which should create a new DB, server config file, and flyweight directory each time). The ~/.github file does exist on the server's Docker image which includes the relevant connection details. I am able to clone impacted repos manually on the running GoCD server Docker container.

As others have noted, the issue seems to be an encoding issue with the "=" in the git clone command that runs on the server. The command should be git clone --branch=master https://repo target_directory but the "=" symbol doesn't seem to get encoded properly and the actual command executed is git clone --branch\u003dmaster https://repo target_directory which leads to the error (\u003d seems to be the unicode encoding of the "=" symbol).