search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.96k stars 17.66k forks source link

cmd/go: add support for Azure DevOps Git repos to go get #28236

Closed MrWako closed 4 years ago

MrWako commented 6 years ago

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.11 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOHOSTARCH=amd64 GOHOSTOS=windows GOOS=windows

What did you do?

I created a few public Azure DevOps repos to illustrate the problems

Then when trying to get the package with the main function I get:

go get -v dev.azure.com/markawakefield/GoGetTest/_git/app
Fetching https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1
Parsing meta tags from https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1 (status code 200)
package dev.azure.com/markawakefield/GoGetTest/_git/app: unrecognized import path "dev.azure.com/markawakefield/GoGetTest/_git/app" (parse https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1: no go-import meta tags ())

I think because go get does not know its a Git repo, and there are no meta tags supplied to indicate that it is. If I try to just get one of the other packages with a .git extension I get:

go get -v dev.azure.com/markawakefield/GoGetTest/_git/stringfmt.git
# cd .; git ls-remote https://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
Logon failed, use ctrl+c to cancel basic credential prompt.
fatal: could not read Username for 'https://dev.azure.com': terminal prompts disabled
# cd .; git ls-remote git+ssh://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
ssh: connect to host dev.azure.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
# cd .; git ls-remote ssh://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
ssh: connect to host dev.azure.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
package dev.azure.com/markawakefield/GoGetTest/_git/stringfmt.git: cannot download, git://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt uses insecure protocol

I think because the .git extension on the URL is not being passed to the git commands, and therefore it hits an authenticated endpoint.

What did you expect to see?

I expected (hoped) to see the packages add to the GOPATH under the dev.azure.com directory.

What did you see instead?

No files added to the GOPATH

I appreciate that this is not a pure issue with the current Go tool chain, more a miss-match between the conventions of Go and Azure DevOps, and has been noted elsewhere including http://ppanyukov.github.io/2017/02/01/golang-with-vsts-repos.html. Please let me know if a proposal is more appropriate.

For other cloud providers (IBM/Apache/OpenStack) there is logic to accomodate these differences and for Azure DevOps its actually a little worse than just the go get issue as even if this command completed the package ends up under a _git directory and so will be ignored in some instances due to the underscore.

Until now we have used Glide for package management which has allowed us to work around the issue but we would now like to adopt versioned packages so the issue is becoming more pressing.

I'll open a PR with a small revision addressing the issue that can be discussed/reviewed.

Thanks

brkerr commented 5 years ago

You'll see the redirect if it is a private project, since we redirect you to sign in. You'll need to follow the PAT and gitconfig approach talked about above.

jpreese commented 5 years ago

ah.. we were trying to avoid having to have PATs in the config of the repository. We'll most likely need to end up going the vanity url server if that is the case then. Thanks!

edit: As a follow up in case it was missed, you do not need a PAT when you include .git in the URL, but that looks all sorts of wonky.

brkerr commented 5 years ago

You'll need a PAT unless git had some sort of credential saved on the computer you're testing it for. Unfortunately private repos will always require some sort of authentication, so until Go comes out with their GoAuth feature (planned for the next release) you'll need either the PAT or you can use SSH + SSH key + gitconfig entry

jpreese commented 5 years ago

Yep! From our tests if we use go get with .git in the url (e.g. https://dev.azure.com/myorg/myproject/_git/myproject.git) it will work with a private repo. The Microsoft login window pops up and asks us to login, which is completely acceptable.

The unfortunate part is that including .git in the url makes your binary from go get include ".git" (e.g. myproject.git.exe) and the imports in your project also must have .git if you reference your own modules (e.g. import dev.azure.com/myorg/myproject/_git/myproject.git/someother/submodule)

If we remove the .git from the url, it doesn't work at all. Which in itself is a little confusing because I would expect the error to say "unauthorized" if a PAT must be in the repository. This flat out says it cannot resolve the repository url.

brkerr commented 5 years ago

Yeah so the goofy part is that Go can see a github or gitlab url and know it's a git repo because that's hard coded in the Go source (the original PR for this issue). For Go to know that an Azure DevOps repo is a git repo you need .git or it won't work. It says it can't resolve because of how Go tries to figure out what kind of repo the url is - it will query the url with "go-get" in the params and look for the meta tag, which will hit the redirect and decide that it can't figure it out.

jpreese commented 5 years ago

Right, I thought the PR was to add the meta tag to the Azure DevOps git page so .git wasn't needed. Maybe I misunderstood.

Adding a go-import meta tag to AzDO should fix that.

brkerr commented 5 years ago

Unfortunately private repos will always require some auth workaround since we weren't able to intercept the page before the redirect.

jpreese commented 5 years ago

I see, thanks for the additional information!

brkerr commented 5 years ago

No problem. We want to work on getting "_git" out of the url and merging the original PR for this issue to remove ".git" as well, but those are coming in behind some other higher priority items on the backlog. Eventually we hope to have parity with existing vanity url services.

prakashpandey commented 5 years ago

Can you please provide us estimated timeline when it will be fixed.

raushan2016 commented 5 years ago

go get go get -v dev.azure.com/foo/bar/_git/abc.git/pkg/xyz worked perfectly fine with PAT token approach. But when i tried doing dep ensure -v -add dev.azure.com/foo/bar/_git/abc.git/pkg/xyz it gets suck and eventually fails with the error

ensure Solve(): no valid source could be created: failed to set up sources from the following URLs: https://dev.azure.com/foo/bar/_git/abc.git : remote repository at https://dev.azure.com/foo/bar/_git/abc.git does not exist, or is inaccessible: : exit status 128 failed to set up sources from the following URLs: ssh://dev.azure.com/foo/bar/_git/abc.git : remote repository at ssh://dev.azure.com/foo/bar/_git/abc.git does not exist, or is inaccessible: : exit status 128 failed to set up sources from the following URLs: git://dev.azure.com/foo/bar/_git/abc.git : remote repository at git://dev.azure.com/foo/bar/_git/abc.git does not exist, or is inaccessible: : exit status 128 failed to set up sources from the following URLs: http://dev.azure.com/foo/bar/_git/abc.git : remote repository at http://dev.azure.com/foo/bar/_git/abc.git does not exist, or is inaccessible: : exit status 128

Does dep ensure not use the PAT token url? Any work around for this issue.

Jonathan34 commented 5 years ago

If you get a 410 gone error with the PAT solution in .gitconfig:

go: finding dev.azure.com/<org>/<project>/_git/<project>.git latest
go: finding dev.azure.com/<org>/<project>/_git/<project>.git/protos latest
go: downloading dev.azure.com/<org>/<project>/_git/<project>.git v0.0.0-20191007171928-c6a8ebcd1fa5
verifying dev.azure.com/<org>/<project>/_git/<project>.git@v0.0.0-20191007171928-c6a8ebcd1fa5: dev.azure.com/<org>/<project>/_git/<project>.git@v0.0.0-20191007171928-c6a8ebcd1fa5: reading https://sum.golang.org/lookup/dev.azure.com/<org>/<project>/_git/<project>.git@v0.0.0-20191007171928-c6a8ebcd1fa5: 410 Gone

Its reading from sum.golang.org, which does not exist there obviously (it s private).

Setting GOPRIVATE (https://golang.org/cmd/go/#hdr-Module_configuration_for_non_public_modules) to dev.azure.com will download the repo in the mod folder.

bcmills commented 5 years ago

@raushan2016, this issue is about support in go get, not dep ensure.

MrWako commented 4 years ago

Sorry, maybe probably off topic too, but if you are here due to authentification issues sync'ing with Azure DevOps repos with go get, e.g. on a build machine then you can setup a simple git credential helper in your gitconfig file: e.g:

[credential "https://dev.azure.com"]
    helper = "/bin/sh /etc/azure-devops-helper.sh"

with azure-devops-helper.sh something like

#!/bin/sh
echo username=$USERNAME
echo password=$AZURE_DEVOPS_TOKEN

where environment variables containing the username and PAT taken. This proved more reliable for us than switching to SSH for azure repos.

bcmills commented 4 years ago

Reopening due to resumed discussion in #41030.

artbegolli commented 4 years ago

@brkerr Is there any update on the state of affairs here? I think if we can get some clarity on the reason as to why adding the <meta> tags would not be feasible on ADOs end we could justify hardcoding the entry in cmd/go/internal/get. Is the main blocker the auth redirect?

For us the last remaining issue is having to specify .git in our paths.

vtbassmatt commented 4 years ago

:wave: hey folks, I'm the product manager for Azure Repos server. I got clued into this thread because users are now reporting that our documentation for using go get with Azure Repos isn't working anymore, such as https://github.com/MicrosoftDocs/azure-devops-docs/issues/9250.

Brandon added the meta tags back in 2019, and we still serve them. (He's moved on from the product team so I won't bug him here.) I'm hitting our Git protocol endpoint with ?go-get=1 appended to it and see this in the resulting <head>:

<meta name="go-import"
      content="dev.azure.com/mattc-demo/Test1/_git/playground git https://dev.azure.com/mattc-demo/Test1/_git/playground">

Does that look correct for a Git URL https://mattc-demo@dev.azure.com/mattc-demo/Test1/_git/playground?

For private projects, you have to present auth before we'll serve any content other than a redirect, and I probably can't change that. It looks like there was an (abandoned) effort to support our URL formats here. Is anyone willing to resurrect that change and land it?

Alternatively, or in addition, if someone can advise me on what we can fix in that doc, I'd also be happy to make the changes.

Thanks!

JoseFMP commented 4 years ago

@vtbassmatt Thank you.

For us, for private repos is not working.

artbegolli commented 4 years ago

@bcmills So based on @vtbassmatt comment - it seems as though the reason why Go is not recognising private ADO repositories as git repos is that an auth redirect is presented before other content and therefore <meta> tags on the repo itself isn't having the desired effect.

So it seems to me as though the two options are:

  1. Hardcoded entry in Go as in this PR or https://github.com/golang/go/pull/41031
  2. tag on auth (which I don't think makes as much sense)
bcmills commented 4 years ago

@artbegolli, @vtbassmatt: the go command should be sending credentials from the user's .netrc file. (See https://golang.org/ref/mod#private-module-proxy-auth, but I notice that we have not yet fully documented the use of .netrc credentials for servers other than module proxies. We should fix that.)

Is that not happening, is the server serving the auth redirect (or failing to serve the go-import <meta> tags) despite valid credentials, or is there something else going on?

JoseFMP commented 4 years ago

Over here we are not using a proxy. Still not working.

We keep getting no go-import meta tags ()) when doing go get.

We tried all the solutions suggested in this thread with no luck.

bcmills commented 4 years ago

@JoseFMP, are you using credentials from a .netrc file, as mentioned in my previous comment?

JoseFMP commented 3 years ago

@bcmills I tried using .netrc as well as other authentication ways here and there, including the ones in https://docs.microsoft.com/en-us/azure/devops/repos/git/go-get?view=azure-devops with no luck so far for private repos. For public works fine.

In private repos I keep getting:

josem@rabbit:~$ go get dev.azure.com/myorg/myproject/pulsaradapter.git@1073
go: downloading dev.azure.com/myorg/myproject/pulsaradapter.git v0.0.0-20201211023249-f2b661c7768f
go get dev.azure.com/myorg/myproject/pulsaradapter.git@1073: dev.azure.com/myorg/myproject/pulsaradapter.git@v0.0.0-20201211023249-f2b661c7768f: verifying module: dev.azure.com/myorg/myproject/pulsaradapter.git@v0.0.0-20201211023249-f2b661c7768f: reading https://sum.golang.org/lookup/dev.azure.com/myorg/!myproject/pulsaradapter.git@v0.0.0-20201211023249-f2b661c7768f: 410 Gone
        server response:
        not found: dev.azure.com/myorg/myproject/pulsaradapter.git@v0.0.0-20201211023249-f2b661c7768f: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/2af3f2ae10b32b5e53fd191e041fc1e3751ac7b215c92b3d0b823915f73813be: exit status 128:
                fatal: unable to connect to dev.azure.com:
                dev.azure.com[0: 13.107.42.20]: errno=Connection refused
                dev.azure.com[1: 2620:1ec:21::20]: errno=Connection refused

401 Gone ===> What is this supposed to mean? Then exist status 128: Unable co to connect to dev.azure.com

JoseFMP commented 3 years ago

Sorry, maybe probably off topic too, but if you are here due to authentification issues sync'ing with Azure DevOps repos with go get, e.g. on a build machine then you can setup a simple git credential helper in your gitconfig file: e.g:

[credential "https://dev.azure.com"]
  helper = "/bin/sh /etc/azure-devops-helper.sh"

with azure-devops-helper.sh something like

#!/bin/sh
echo username=$USERNAME
echo password=$AZURE_DEVOPS_TOKEN

where environment variables containing the username and PAT taken. This proved more reliable for us than switching to SSH for azure repos.

Silly question here. What's supposed to be in USERNAME var ?

JoseFMP commented 3 years ago

Solved now... needed to tell explicitly in GOPRIVATE to not to check checksums of dev.azure.com .

It seems it was really only that.