search

keybase / client

Keybase Go Library, Client, Service, OS X, iOS, Android, Electron
BSD 3-Clause "New" or "Revised" License
8.89k stars 1.23k forks source link

git-remote-keybase error: (1) malformed refspec, separators are wrong #11018

Open randarnold opened 6 years ago

randarnold commented 6 years ago

I'm trying to see if I can get keybase git working with buildbot

When the GitPoller is invoked I get the following error

2018-03-21 18:39:00-0700 [-] command fetch ['keybase://team/spotlightlabs.git/spotlight-ios', '+bb-test:refs/buildbot/keybase%3A%2F%2Fteam%2Fspotlightlabs.git%2Fspotlight-ios/bb-test', '+develop:refs/buildbot/keybase%3A%2F%2Fteam%2Fspotlightlabs.git%2Fspotlight-ios/develop', '+master:refs/buildbot/keybase%3A%2F%2Fteam%2Fspotlightlabs.git%2Fspotlight-ios/master'] in /Volumes/buildbot/buildmaster/gitpoller-work on repourl keybase://team/spotlightlabs.git/spotlight-ios failed with exit code 128: Initializing Keybase... done. Syncing with Keybase... done. git-remote-keybase error: (1) malformed refspec, separators are wrong

My guess is that the escapes are a problem?

macOS 10.13.3, buildbot 1.1, keybase 1.0.45

strib commented 6 years ago

Can you please do a keybase log send and we can take a look at exactly what refspec is being passed in. I'm not familiar with gitpoller.

randarnold commented 6 years ago

Done, log id 8d1d77bc8d927929f207e11c

strib commented 6 years ago

The problem seems to be that the git remote name you've assigned to the keybase git URL is the actual URL itself, i.e., "keybase://team/spotlightlabs.git/spotlight-ios". Remote names are usually just letters/numbers, like "origin" or "keybase". Again, I'm not sure about GitPoller, but if you're on the command line, usually you would set a remote name like this:

git remote add keybase keybase://team/spotlightlabs.git/spotlight-ios

And then you can just refer to "keybase" as the remote name.

randarnold commented 6 years ago

Thank you, I'll give that a shot

randarnold commented 6 years ago

OK, I've learned a lot, but not enough... buildbot is a CI built in Python, gitpoller.py looks for changes on the repository. https://github.com/buildbot/buildbot/blob/master/master/buildbot/changes/gitpoller.py

GitPoller takes as arguments a reprourl and branches http://docs.buildbot.net/current/manual/cfg-changesources.html#gitpoller

It does nothing with remotes, rather builds a git fetch command with the url and refspecs

This works with a github url, but not with a keybase url (keybase url does work from the cmd line, although not with the generated refspec)

github: '+develop:refs/buildbot/git%40github.com%3ASpotlightMediaLabs%2FSpotlight.git/develop' keybase: '+develop:refs/buildbot/keybase%3A%2F%2Fteam%2Fspotlightlabs.git%2Fspotlight-ios/develop'

github is happy as a clam, keybase returns code 128 git-remote-keybase error: (1) malformed refspec, separators are wrong

obviously my previous guess about the escapes is not the problem (with no escapes everything fails)

I tried keybase::keybase://… that did not work either

I couldn't find the generated error in the source of git-remote-keybase, so I don't who actually generates that error.

Any other thoughts, I'm game :-)

strib commented 6 years ago

The error is coming from here, in the go-git library we use: https://github.com/keybase/kbfs/blob/eecab64a434e2f6a017e69659ef11e3545b4fbe8/vendor/gopkg.in/src-d/go-git.v4/config/refspec.go#L35

I guess it doesn't like there being more than one ":" character in the refspec (i.e., the one after "develop", and the %3A after "keybase" in your refspec above). It seems this is a correct restriction to me. On the command line, this gives an error:

$ git remote add git@github:keybase/test.git keybase://private/strib/test
fatal: 'git@github:keybase/test.git' is not a valid remote name
$ git checkout -b keybase://private/strib/test
fatal: 'keybase://private/strib/test' is not a valid branch name.

I would guess that whatever git server your build bot is running incorrectly allows colons in branch names, but our strict checks in git-remote-keybase correctly prevent this on the client side. I'm not 100% sure though.

randarnold commented 6 years ago

Ok, I think I see the error here…

From git-scm.com

The format of a parameter is an optional plus +, followed by the source , followed by a colon :, followed by the destination ref . The colon can be omitted when is empty. is typically a ref, but it can also be a fully spelled hex object name.

Not sure what the point of the option plus + is, but after the there is the colon : if there is a

breaking down my example '+develop:refs/buildbot/keybase://team/spotlightlabs.git/spotlight-ios/develop' +

develop : refs/buildbot/keybase://team/spotlightlabs.git/spotlight-ios/develop Two colons, both valid (and if I understand correctly neccessary), as the here needs the : for the transport, which is also spelled out in refspec.go, however refspect.go is assuming remotes, which are not used by buildbot. So 1 colon by remote, 2 by url (ok, so probably not that funny…) I'm not sure where the fix is here, but as is I don't think I can use keybase-git without remotes… I don't think I can use remotes without a rewrite of the buildbot scripts (not going to go there).
babymotte commented 5 years ago

Is there any progress on this? This issue also prevents Jenkins pipelines from being able to build projects hosted in a keybase repo, which is a real bummer.

babymotte commented 5 years ago

Here's the error I get in Jenkins:

> git fetch --no-tags --force --progress keybase://team/bbmsoft_net.jam/jam +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from keybase://team/bbmsoft_net.jam/jam
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
    at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.plugins.git.GitException: Command "git fetch --no-tags --force --progress keybase://team/bbmsoft_net.jam/jam +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: Initializing Keybase... done.
Syncing with Keybase... done.
git-remote-keybase error: (1) malformed refspec, separators are wrong

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2099)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1791)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:79)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:472)
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
    ... 11 more

As far as I can tell, the fetch command is perfectly valid ('git fetch --help' explicitly states that fetch is supposed to work on both named repos and raw URLs).