Git Fetch does not work with Azure DevOps repos

jessesuen commented 5 years ago

We use go-git in our K8s GitOps Deployment tool and users are reporting issues with Azure DevOps git repos. It appears that go-git does not work with Azure DevOps git repos. Fetch fails with the following error: unexpected client error: unexpected requesting "https://jsuen0437@dev.azure.com/jsuen0437/jsuen/_git/jsuen/git-upload-pack" status code: 400

The go-git call we are making are as follows:

    err = repo.Fetch(&git.FetchOptions{
        RemoteName: git.DefaultRemoteName,
        Tags:       git.AllTags,
        Force:      true,
    })

The equivalent git CLI command will succeed git fetch origin --tags --force.

I traced the error down to the upload_pack.go::UploadPack() when it is making the HTTP request:

func (s *upSession) UploadPack(
    ctx context.Context, req *packp.UploadPackRequest,
) (*packp.UploadPackResponse, error) {

    if req.IsEmpty() {
        return nil, transport.ErrEmptyUploadPackRequest
    }

    if err := req.Validate(); err != nil {
        return nil, err
    }

    url := fmt.Sprintf(
        "%s/%s",
        s.endpoint.String(), transport.UploadPackServiceName,
    )

    content, err := uploadPackRequestToReader(req)
    if err != nil {
        return nil, err
    }

    res, err := s.doRequest(ctx, http.MethodPost, url, content)  // <<<<< fails here
    if err != nil {
        return nil, err
    }
...
}

The values to the doRequest() call are:

url: https://jsuen0437@dev.azure.com/jsuen0437/jsuen/_git/jsuen/git-upload-pack
content: 004cwant 58594acf362b1452349e2a15163d3357b9160663 side-band-64k no-progress
00000009done

I tested with the following go-git versions:

v4.7.0
v4.9.1

EDIT: updated to note that this fails even with public repos. Please feel free to use: https://jsuen0437@dev.azure.com/jsuen0437/jsuen/_git/jsuen for testing this.

jessesuen commented 5 years ago

Another data point is that another user reports that SSH also fails with azure: remote: Command git-upload-pack '/YYY/PROJECT/REPONAME' is not in expected format.

mcuadros commented 5 years ago

Its related with https://github.com/src-d/go-git/issues/335, looks like MS didn't implement, the basic protocol on his services. And only implements multi-ack protocol.

In short, we are not going to implement multi-ack (at least by @src-d), instead, we are already working on the implementation of the new git wire protocol, the v2. Can you test against azure, a git client with v2? (https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html)

jessesuen commented 5 years ago

instead, we are already working on the implementation of the new git wire protocol, the v2. Can you test against azure, a git client with v2? (https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html)

@mcuadros Sure I'd be happy to! However, I'm not sure what exactly you mean by testing "a git client v2". Is there a pointer to some example code which is doing this?

mcuadros commented 5 years ago

I mean using, git with v2 protocol against Azure. You have an example in the link I pasted

git clone https://chromium.googlesource.com/chromium/src.git
cd src
export GIT_TRACE_PACKET=1

# Using protocol version 2
git -c protocol.version=2 fetch --no-tags origin master

jessesuen commented 5 years ago

Yes this seems to work:

$ git clone https://jsuen0437@dev.azure.com/jsuen0437/jsuen/_git/jsuen
Cloning into 'jsuen'...
remote: Azure Repos
remote: Found 237 objects to send. (22 ms)
Receiving objects: 100% (237/237), 46.04 KiB | 7.67 MiB/s, done.
Resolving deltas: 100% (120/120), done.

$ cd jsuen/

$ export GIT_TRACE_PACKET=1

$ git -c protocol.version=2 fetch --no-tags origin master
23:21:13.422461 pkt-line.c:80           packet:          git< # service=git-upload-pack
23:21:13.422819 pkt-line.c:80           packet:          git< 0000
23:21:13.422825 pkt-line.c:80           packet:          git< 58594acf362b1452349e2a15163d3357b9160663 HEAD\0 multi_ack thin-pack side-band side-band-64k no-progress multi_ack_detailed no-done shallow allow-tip-sha1-in-want symref=HEAD:refs/heads/master
23:21:13.422838 pkt-line.c:80           packet:          git< 58594acf362b1452349e2a15163d3357b9160663 refs/heads/master
23:21:13.422842 pkt-line.c:80           packet:          git< 0000
From https://dev.azure.com/jsuen0437/jsuen/_git/jsuen
 * branch            master     -> FETCH_HEAD

jessesuen commented 5 years ago

What's the timeframe for v2? We'd just like to understand if we should change the implementation to fork/exec git CLI to workaround this.

delaman commented 5 years ago

I am also experiencing this issue with the following setup:

go 1.12

require (
    github.com/gliderlabs/ssh v0.2.2 // indirect
    github.com/google/go-cmp v0.3.0 // indirect
    github.com/kr/pty v1.1.5 // indirect
    github.com/stretchr/objx v0.2.0 // indirect
    golang.org/x/crypto v0.0.0-20190618222545-ea8f1a30c443 // indirect
    golang.org/x/net v0.0.0-20190619014844-b5b0513f8c1b // indirect
    golang.org/x/sys v0.0.0-20190620070143-6f217b454f45 // indirect
    golang.org/x/text v0.3.2 // indirect
    golang.org/x/tools v0.0.0-20190620154339-431033348dd0 // indirect
    gopkg.in/src-d/go-git.v4 v4.12.0
)

Any word on when this issue will be resolved?

memark commented 4 years ago

What's the timeframe for v2? We'd just like to understand if we should change the implementation to fork/exec git CLI to workaround this.

He tried your command as requested (v2 against Azure) and it worked. Do you have a response? :-)

Is there any timeframe for v2, @mcuadros?

src-d / go-git

Git Fetch does not work with Azure DevOps repos #1058