search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
121.36k stars 17.38k forks source link

cmd/go: get fails to provide sensible error message for private vcs repos #25982

Open vasilevp opened 6 years ago

vasilevp commented 6 years ago

What did you do?

I have a project that imports a private git repository. When setting it up with vgo get or similar commands, the resolving process will stop abruptly, not writing anything to the disk.

The command succeeds if git credentials are properly set up (*_ASKPASS, global config or the repo is already cloned with the right config, etc.).

What did you expect to see?

An error message, or some indication of what went wrong

What did you see instead?

No error message, just an exit code 1.

System details

go version go1.10.3 linux/amd64 vgo:2018-02-20.1
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/exploser/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/exploser/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build534162883=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version go1.10.3 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.10.3
uname -sr: Linux 4.16.14-2-MANJARO
LSB Version:    n/a
Distributor ID: ManjaroLinux
Description:    Manjaro Linux
Release:    17.1.10
Codename:   Hakoila
/usr/lib/libc.so.6: GNU C Library (GNU libc) stable release version 2.27.
gdb --version: GNU gdb (GDB) 8.1
vito-c commented 6 years ago

I was playing around with modules and ran into an error issue similar to the one in this ticket. myprivate/repo is a private repo. I have all of my ssh keys setup correctly and I can clone the repo myself.

Here is the error:

go: converting vendor/manifest: stat github.com/myprivate/repo@1234sha: git fetch -f --depth=1 origin 1234sha:refs/dummy in /Users/vito-c/code/go/src/mod/cache/vcs/456sha: exit status 128:
    fatal: could not read Username for 'https://github.com': terminal prompts disabled

Should I open a new issue for this? I believe it was trying to authenticate over https not via ssh because in my vendor/manifest there is https://github.com/mypriavet/repo If there is any workaround you want me to try let me know.

wayneashleyberry commented 5 years ago

This also failed for me while running 1.11beta2, not sure if it's related.

$ go1.11beta2 build
go: finding github.com/org/repo v6.9.0
go: errors parsing go.mod:
/foo/bar/go.mod:31: invalid module version "v6.9.0": unknown revision v6.9.0

Where github.com/org/repo is a private repo which can get git clone'd just fine.

wayneashleyberry commented 5 years ago

Just tried initialising a project that uses dep with beta 3.

$ go1.11beta3 mod init
go: creating new go.mod: module github.com/overhq/over-stories-api
go: copying requirements from Gopkg.lock
go: converting Gopkg.lock: stat github.com/overhq/go-lib@v7.1.2: unknown revision v7.1.2
$ go1.11beta3 build main.go
go: finding github.com/gobuffalo/makr v1.1.0
go: finding github.com/joho/godotenv v1.2.0
go: finding github.com/gobuffalo/pop v0.0.0-20180726201638-c0faa0f7edf8
go: finding github.com/googleapis/gax-go v2.0.0+incompatible
go: finding github.com/sergi/go-diff v1.0.0
go: finding github.com/gobuffalo/uuid v2.0.0+incompatible
go: finding github.com/gobuffalo/packr v1.12.0
go: finding github.com/go-chi/chi v3.3.2+incompatible
go: finding github.com/russross/blackfriday v0.0.0-20180428102519-11635eb403ff
go: finding github.com/gobuffalo/fizz v1.0.2
go: finding github.com/spf13/cobra v0.0.3
go: finding github.com/markbates/going v1.0.0
go: finding github.com/francoispqt/gojay v0.0.0-20180723134641-7dc0f34a209c
go: finding github.com/golang/protobuf v1.1.0
go: finding github.com/shurcooL/github_flavored_markdown v0.0.0-20180602233135-8913699a52e3
go: finding github.com/gobuffalo/packr v1.11.1
go: finding github.com/gobuffalo/validate v1.0.0
go: finding github.com/markbates/inflect v1.0.0
go: finding github.com/mitchellh/go-homedir v0.0.0-20180523094522-3864e76763d9
go: finding github.com/gobuffalo/tags v0.0.0-20180702174034-96c7cdfcadcb
go: finding github.com/go-gorp/gorp v0.0.0-20180410155428-6032c66e0f5f
go: finding github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
go: finding github.com/spf13/pflag v1.0.1
go: finding github.com/go-redis/redis v6.13.2+incompatible
go: finding github.com/microcosm-cc/bluemonday v1.0.0
go: finding github.com/shurcooL/octicon v0.0.0-20180602230221-c42b0e3b24d9
go: finding github.com/mattn/go-isatty v0.0.3
go: finding golang.org/x/sys v0.0.0-20180616030259-6c888cc515d3
go: finding google.golang.org/api v0.0.0-20180726000515-082d5fa4f1f0
go: finding github.com/go-sql-driver/mysql v1.4.0
go: finding go.uber.org/zap v1.9.0
go: finding github.com/markbates/grift v1.0.0
go: finding google.golang.org/genproto v0.0.0-20180726180014-2a72893556e4
go: finding github.com/serenize/snaker v0.0.0-20171204205717-a683aaf2d516
go: finding gopkg.in/check.v1 v1.0.0-20161208181325-20d25e280405
go: finding gopkg.in/square/go-jose.v2 v2.1.7
go: finding golang.org/x/text v0.3.0
go: finding google.golang.org/appengine v1.1.0
go: finding golang.org/x/sys v0.0.0-20180715085529-ac767d655b30
go: finding github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
go: finding github.com/pkg/errors v0.8.0
go: finding github.com/mattn/go-sqlite3 v1.9.0
go: finding github.com/mattn/go-colorable v0.0.9
go: finding github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95
go: finding github.com/gobuffalo/validate v2.0.0+incompatible
go: finding cloud.google.com/go v0.23.0
go: finding golang.org/x/oauth2 v0.0.0-20180724155351-3d292e4d0cdc
go: finding go.uber.org/multierr v1.1.0
go: finding github.com/inconshreveable/mousetrap v1.0.0
go: finding github.com/markbates/validate v1.0.0
go: finding dmitri.shuralyov.com/text/kebabcase v0.0.0-20180217051803-40e40b42552a
go: finding github.com/gobuffalo/envy v1.6.3
go: finding github.com/satori/go.uuid v1.2.0
go: finding github.com/gobuffalo/plush v0.0.0-20180619220412-2ec029f415ed
go: finding go.uber.org/atomic v1.3.2
go: finding golang.org/x/sys v0.0.0-20180724212812-e072cadbbdc8
go: finding github.com/mattn/anko v0.0.5
go: finding github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
go: finding github.com/kelseyhightower/envconfig v1.3.0
go: finding github.com/fatih/color v1.7.0
go: finding github.com/stretchr/testify v1.2.2
go: finding golang.org/x/net v0.0.0-20180611182652-db08ff08e862
go: finding google.golang.org/grpc v1.13.0
go: finding github.com/cockroachdb/cockroach-go v0.0.0-20180212155653-59c0560478b7
go: finding github.com/shurcooL/highlight_go v0.0.0-20170515013102-78fb10f4a5f8
go: finding golang.org/x/net v0.0.0-20180712202826-d0887baf81f4
go: finding github.com/gobuffalo/uuid v0.0.0-20180207211247-3a9fb6c5c481
go: finding golang.org/x/net v0.0.0-20180724234803-3673e40ba225
go: finding github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e
go: finding github.com/fatih/structs v1.0.0
go: finding github.com/shurcooL/graphql v0.0.0-20180514000029-62c9ce094e75
go: finding github.com/stretchr/objx v0.1.1
go: finding golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
go: finding github.com/newrelic/go-agent v2.0.0+incompatible
go: finding github.com/gobuffalo/pop v0.0.0-20180710054625-9936c00d4230
go: finding github.com/lib/pq v0.0.0-20180523175426-90697d60dd84
go: finding gopkg.in/yaml.v2 v2.2.1
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding github.com/gobuffalo/plush v0.0.0-20180724100351-726a3d7f08d2
go: finding golang.org/x/crypto v0.0.0-20180723164146-c126467f60eb
go: finding github.com/jmoiron/sqlx v0.0.0-20180614180643-0dae4fefe7c0
go: finding github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041
go: finding github.com/inconshreveable/mousetrap v0.0.0-20141017200713-76626ae9c91c
go: finding go.opencensus.io v0.14.0
go: finding github.com/microcosm-cc/bluemonday v0.0.0-20180327211928-995366fdf961
go: finding github.com/fatih/structs v0.0.0-20170103081050-a720dfa8df58
go: finding github.com/gobuffalo/validate v0.0.0-20180210182920-42d8db6e06e6
go: finding github.com/gobuffalo/tags v0.0.0-20180320140155-82cd7696c84a
go: downloading github.com/spf13/cobra v0.0.3
go: downloading github.com/kelseyhightower/envconfig v1.3.0
go: downloading github.com/joho/godotenv v1.2.0
go: downloading github.com/go-redis/redis v6.13.2+incompatible
go: downloading github.com/pkg/errors v0.8.0
go: downloading github.com/gobuffalo/pop v0.0.0-20180726201638-c0faa0f7edf8
go: downloading github.com/francoispqt/gojay v0.0.0-20180723134641-7dc0f34a209c
go: downloading github.com/satori/go.uuid v1.2.0
go: downloading google.golang.org/api v0.0.0-20180726000515-082d5fa4f1f0
go: downloading cloud.google.com/go v0.23.0
go: downloading github.com/go-chi/chi v3.3.2+incompatible
go: downloading github.com/gobuffalo/uuid v2.0.0+incompatible
go: downloading github.com/markbates/inflect v1.0.0
go: downloading github.com/gobuffalo/fizz v1.0.2
go: downloading github.com/spf13/pflag v1.0.1
go: downloading github.com/fatih/color v1.7.0
go: downloading github.com/cockroachdb/cockroach-go v0.0.0-20180212155653-59c0560478b7
go: downloading github.com/lib/pq v0.0.0-20180523175426-90697d60dd84
go: downloading github.com/jmoiron/sqlx v0.0.0-20180614180643-0dae4fefe7c0
go: downloading github.com/markbates/going v1.0.0
go: downloading github.com/go-sql-driver/mysql v1.4.0
go: downloading github.com/gobuffalo/packr v1.12.0
go: downloading github.com/gobuffalo/makr v1.1.0
go: downloading github.com/mattn/go-colorable v0.0.9
go: downloading github.com/mattn/go-isatty v0.0.3
go: downloading github.com/gobuffalo/envy v1.6.3
go: downloading github.com/gobuffalo/plush v0.0.0-20180724100351-726a3d7f08d2
go: downloading github.com/go-gorp/gorp v0.0.0-20180410155428-6032c66e0f5f
go: downloading github.com/gobuffalo/validate v2.0.0+incompatible
go: downloading github.com/gobuffalo/tags v0.0.0-20180702174034-96c7cdfcadcb
go: downloading gopkg.in/yaml.v2 v2.2.1
go: downloading github.com/shurcooL/github_flavored_markdown v0.0.0-20180602233135-8913699a52e3
go: downloading github.com/shurcooL/highlight_go v0.0.0-20170515013102-78fb10f4a5f8
go: downloading golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
go: downloading github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
go: downloading github.com/microcosm-cc/bluemonday v1.0.0
go: downloading golang.org/x/net v0.0.0-20180724234803-3673e40ba225
go: downloading github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95
go: downloading github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
go: downloading github.com/serenize/snaker v0.0.0-20171204205717-a683aaf2d516
go: downloading github.com/russross/blackfriday v0.0.0-20180428102519-11635eb403ff
go: downloading github.com/shurcooL/octicon v0.0.0-20180602230221-c42b0e3b24d9
go: downloading github.com/fatih/structs v1.0.0
go: downloading github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
go: downloading github.com/sergi/go-diff v1.0.0
go: downloading github.com/googleapis/gax-go v2.0.0+incompatible
go: downloading go.opencensus.io v0.14.0
go: downloading google.golang.org/grpc v1.13.0
go: downloading google.golang.org/genproto v0.0.0-20180726180014-2a72893556e4
go: downloading github.com/golang/protobuf v1.1.0
go: downloading golang.org/x/text v0.3.0
go: downloading golang.org/x/oauth2 v0.0.0-20180724155351-3d292e4d0cdc
go: import "github.com/overhq/over-stories-api/cmd" ->
    import "github.com/overhq/over-stories-api/cmd/app" ->
    import "github.com/overhq/go-lib/pkg/log": cannot find module providing package github.com/overhq/go-lib/pkg/log
go: import "github.com/overhq/over-stories-api/cmd" ->
    import "github.com/overhq/over-stories-api/cmd/app" ->
    import "github.com/overhq/over-stories-api/pkg/app" ->
    import "github.com/overhq/go-lib/pkg/middleware": cannot find module providing package github.com/overhq/go-lib/pkg/middleware
go: import "github.com/overhq/over-stories-api/cmd" ->
    import "github.com/overhq/over-stories-api/cmd/app" ->
    import "github.com/overhq/over-stories-api/pkg/app" ->
    import "github.com/overhq/go-lib/pkg/servingurl": cannot find module providing package github.com/overhq/go-lib/pkg/servingurl
go: import "github.com/overhq/over-stories-api/cmd" ->
    import "github.com/overhq/over-stories-api/cmd/app" ->
    import "github.com/overhq/over-stories-api/pkg/app" ->
    import "github.com/overhq/over-stories-api/pkg/handler" ->
    import "github.com/overhq/go-lib/pkg/datetime": cannot find module providing package github.com/overhq/go-lib/pkg/datetime
go: import "github.com/overhq/over-stories-api/cmd" ->
    import "github.com/overhq/over-stories-api/cmd/app" ->
    import "github.com/overhq/over-stories-api/pkg/app" ->
    import "github.com/overhq/over-stories-api/pkg/handler" ->
    import "github.com/overhq/go-lib/pkg/errors": cannot find module providing package github.com/overhq/go-lib/pkg/errors
go: import "github.com/overhq/over-stories-api/cmd" ->
    import "github.com/overhq/over-stories-api/cmd/app" ->
    import "github.com/overhq/over-stories-api/pkg/app" ->
    import "github.com/overhq/over-stories-api/pkg/cache" ->
    import "github.com/overhq/over-stories-api/pkg/model" ->
    import "github.com/overhq/go-lib/pkg/strconv": cannot find module providing package github.com/overhq/go-lib/pkg/strconv
myitcv commented 5 years ago

Just to confirm @wayneashleyberry you're expecting that to succeed because a git clone for, say, github.com/overhq/over-stories-api succeeds?

Are you using ssh-based authentication here or some other?

wayneashleyberry commented 5 years ago

@myitcv hi 👋

you're expecting that to succeed because a git clone for, say, github.com/overhq/over-stories-api succeeds?

absolutely correct

ssh-based authentication here or some other?

ssh auth

cat ~/.ssh/config
Host *
 AddKeysToAgent yes
 UseKeychain yes
 IdentityFile ~/.ssh/id_rsa
myitcv commented 5 years ago

Hi indeed @wayneashleyberry!

Do you also have an insteadOf config for git? Something like url.git@github.com:.insteadof=https://github.com/?

wayneashleyberry commented 5 years ago

nope, what I pasted above is the only config I use for github

wayneashleyberry commented 5 years ago

Just read https://golang.org/doc/faq#git_https and that fixes things for me! Thanks for pointing me in the right direction @myitcv

myitcv commented 5 years ago

👍 @wayneashleyberry - we could/should also add to that FAQ section details of how to use keychains (on any platform).

myitcv commented 5 years ago

So just to summarise what I think the action is here, we need any of the go commands that end up doing anything with a VCS remote (via the corresponding tool), e.g. git ls-remote -q https://github.com/overhq/over-stories-api, to print to stderr the output from the tool in case of a non-zero exit code.

go get is not currently silent (I'll update the issue title) it's just that its output is misleading (comment true as of a9dcbab0fd)

So, taking the example of github.com/overhq/over-stories-api, the current behaviour is:

$ go get github.com/overhq/over-stories-api
go get github.com/overhq/over-stories-api: cannot find module providing package github.com/overhq/over-stories-api

We should instead see something like:

$ go get github.com/overhq/over-stories-api
remote: Repository not found.
fatal: repository 'https://github.com/overhq/over-stories-api/' not found
rsc commented 5 years ago

@bradfitz tells me that the "silent exit 1" fix fixed private repos that worked with "old go get". We can address even better support in Go 1.12.

rsc commented 5 years ago

In non-module mode I see:

$ go get github.com/overhq/over-stories-api
# cd .; git clone https://github.com/overhq/over-stories-api /Users/rsc/src/github.com/overhq/over-stories-api
Cloning into '/Users/rsc/src/github.com/overhq/over-stories-api'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
package github.com/overhq/over-stories-api: exit status 128
$

In module mode I see:

$ go get github.com/overhq/over-stories-api
go get github.com/overhq/over-stories-api: git ls-remote -q origin in /Users/rsc/pkg/mod/cache/vcs/35b63ced1d50599254b07b2a3912fa8f48350519db24e14283e93ce370969e20: exit status 128:
    fatal: could not read Username for 'https://github.com': terminal prompts disabled
$

It's possible that we should recognize this failure (maybe 'terminal prompts disabled') and print a pointer to https://golang.org/doc/faq#git_https.

rsc commented 5 years ago

This was a release blocker for Go 1.11 when it was a silent exit, and we fixed that. The extra doc pointer does not block Go 1.12

gopherbot commented 5 years ago

Change https://golang.org/cl/150777 mentions this issue: cmd/go: link to the FAQ for errors that indicate private VCS repos. Updates #25982

kalexmills commented 3 years ago

Commenting here since I just opened a duplicate.

Could the sorts of issues we see in this ticket be due in part to how the current code is not handling the r.refsErr after line 298 in the below snippet?

https://github.com/golang/go/blob/f1980efb92c011eab71aa61b68ccf58d845d1de7/src/cmd/go/internal/modfetch/codehost/git.go#L298-L336

It looks like refsErr is being set in loadRefs here https://github.com/golang/go/blob/f1980efb92c011eab71aa61b68ccf58d845d1de7/src/cmd/go/internal/modfetch/codehost/git.go#L179-L198

So ISTM that we should check if it was set after every call to r.loadRefs. Of course, that may only be part of the problem. I'm open to opening a PR to resolve the above if it would be productive.

gopherbot commented 3 years ago

Change https://golang.org/cl/284974 mentions this issue: codehost: return errors after loading references during calls to