search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.46k stars 17.59k forks source link

cmd/go: git remote (Gerrit) did not send all necessary objects #31399

Closed ukai closed 5 years ago

ukai commented 5 years ago

What version of Go are you using (go version)?

$ go version
go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/ukai/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr/local/google/home/ukai/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/google-golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build741696436=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ ls /tmp/go
ls: cannot access '/tmp/go': No such file or directory
$ mkdir /tmp/go
$ GOPATH=/tmp/go GO111MODULE=on go install go.chromium.org/goma/server/cmd/remoteexec_proxy

What did you expect to see?

go modules downloaded and binary is built.

What did you see instead?

...
go: finding grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
go: google.golang.org/api@v0.0.0-20181030000543-1d582fd0359e: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: bad object c7964aab01c00a07d0cc6216a78a5570459cebfd
        error: remote did not send all necessary objects
..
go: error loading module requirements

I could fix it by

$ cd /tmp/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0; git fetch --unshallow

then go install works as expected.

https://bugs.chromium.org/p/chromium/issues/detail?id=951546

agnivade commented 5 years ago

@bcmills @jayconrod

KillerX commented 5 years ago

I can confirm that this is happening to me as well. It is not 100% constant. My docker builds experience it about 98% of the time.

go: google.golang.org/api@v0.0.0-20180910000450-7ca32eb868bf: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 1:
    fatal: bad object c7964aab01c00a07d0cc6216a78a5570459cebfd
    error: https://code.googlesource.com/google-api-go-client did not send all necessary objects

A git fetch seems to fix it.

Same issues with go 1.11.5, 1.11.8, 1.12.3 in Docker (golang:1.12.3), and installed locally on Ubuntu.

Local git:

➜   git --version 
git version 2.19.1

I also less frequently get this issue:

go: cloud.google.com/go@v0.0.0-20180131234750-2de512d2700d: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/b0e27935eb83c1d7843713bafab507e95768b550f0552cb42d9f41e5fd9c8375: exit status 1:
        fatal: bad object 3fdddcac5af93820a98679aa0008db7413a13265
        error: https://code.googlesource.com/gocloud did not send all necessary objects

EDIT:

It appears that this patch resolves the issue for my builds:

--- a/packages/go/packagename/go.mod
+++ b/packages/go/packagename/go.mod
@@ -7,7 +7,7 @@ require (
        redacted/logging v0.0.0
        golang.org/x/net v0.0.0-20190320064053-1272bf9dcd53 // indirect
        golang.org/x/oauth2 v0.0.0-20190319182350-c85d3e98c914 // indirect
-       google.golang.org/api v0.3.1
+       google.golang.org/api  v0.0.0-20180603000442-8e296ef26005
        google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107
        google.golang.org/grpc v1.19.1
 )
jayconrod commented 5 years ago

Reproduced locally with the same error.

Also seen with the same instructions as above:

go: golang.org/x/exp@v0.0.0-20190121172915-509febef88a4: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/tmp.yf9bsDvK/pkg/mod/cache/vcs/abc0aaaae9fbb1a906994bfb84d444789fc395ac48e2b740ecbbfeebd3047a32: exit status 128:
    fatal: remote error: authentication error

Perhaps Gerrit is acting up again?

cc @heschik @andybons @hakim @dmitshur

jayconrod commented 5 years ago

I've traced this to a sequence of Git commands that reproduces this issue outside of Go. The Gerrit team has been notified and is working to resolve the issue.

mr-tron commented 5 years ago

Is it some workaround for the problem?

heschi commented 5 years ago

The latest we have from the Gerrit team is that this might be a bug specific to the content of the google.golang.org/api repository. As suggested above, a git fetch should work around it. I think fetching a different version might help too.

mr-tron commented 5 years ago

Yes. Second running also helps, but it very ugly hack. I added vendor dir to my projects. It looks like better workaround.

draftcode commented 5 years ago

Hello, this is the team that manages googlesource.com. *.googlesource.com is hosted by using JGit, and we believe that there's a bug in upstream JGit that cannot handle unshallow git-fetch. We recognize that this is reproducible easily, but we cannot spot the buggy code in the JGit. If there's an option to do a full clone always instead of partial clone, that can be a workaround.

draftcode commented 5 years ago

We identified the root cause and the fix is uploaded at https://git.eclipse.org/r/c/140463/. Our in-house JGit comitters are already gone home, so this will be reviewed tomorrow at best. If everything goes well and our SREs are happy about a Friday release, we might be able to cherry-pick this fix tomorrow. The worst case ETA would be next week.

jrn commented 5 years ago

https://git.eclipse.org/r/c/140463/

That did the trick. If it happens again, we want to know.

chmking commented 5 years ago

@jrn @draftcode

Everything was working this morning but I've seen a regression in the past 10 mins:

12-Apr-2019 13:21:54 | go: google.golang.org/api@v0.0.0-20181021000519-a2651947f503: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 1:
12-Apr-2019 13:21:54 | fatal: bad object 19e022d8cf43ce81f046bae8cc18c5397cc7732f
12-Apr-2019 13:21:54 | error: https://code.googlesource.com/google-api-go-client did not send all necessary objects
12-Apr-2019 13:27:12 | verifying google.golang.org/api@v0.3.1/go.mod: checksum mismatch
12-Apr-2019 13:27:12 | downloaded: h1:4mhQ8q/RsB7i+udVvVy5NUi08OU8ZlA0gRVgrF7VFY0=
12-Apr-2019 13:27:12 | go.sum:     h1:6wY9I6uQWHQ8EM57III9mq/AjF+i8G65rmVagqKMtkk=
mr-tron commented 5 years ago

Still doesnt work for me.

jayconrod commented 5 years ago

I've tried to reproduce this again, and it's now working for me.

@chmking @mr-tron Are you able to reproduce this with a clean module cache using the original instructions at the top of the issue? Could you post the output of go install -x go.chromium.org/goma/server/cmd/remoteexec_proxy with the failure? The -x flag should show the git commands run in each vcs cache, so that will help us isolate the issue.

chmking commented 5 years ago

@jayconrod This is happening on our CI machines running the build in a docker without a go mod cache. It worked this morning without fail many times until it started failing at the time of my post. Now the issues is intermittent.

I can try to replicate it for you later.

draveness commented 5 years ago

I met the same problem when using go get https://github.com/kubernetes/client-go/issues/590

$ go get -u -v k8s.io/client-go
Fetching https://cloud.google.com/go?go-get=1
Parsing meta tags from https://cloud.google.com/go?go-get=1 (status code 200)
get "cloud.google.com/go": found meta tag get.metaImport{Prefix:"cloud.google.com/go", VCS:"git", RepoRoot:"https://code.googlesource.com/gocloud"} at https://cloud.google.com/go?go-get=1
go: finding cloud.google.com/go v0.0.0-20160913182117-3b1ae45394a2
go: cloud.google.com/go@v0.0.0-20160913182117-3b1ae45394a2: git -c protocol.version=0 fetch --unshallow -f https://code.googlesource.com/gocloud refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/draveness/go/pkg/mod/cache/vcs/b0e27935eb83c1d7843713bafab507e95768b550f0552cb42d9f41e5fd9c8375: exit status 128:
    fatal: bad object a5913b3f7deecba45e98ff33cefbac4fd204ddd7
    error: remote did not send all necessary objects
go: error loading module requirements

$ git --version
git version 2.21.0
hazcod commented 5 years ago

Is there a workaround for go get? I am unable to develop. Edit: I skipped over the workaround.

agnivade commented 5 years ago

Please see the first comment for the workaround.

hazcod commented 5 years ago

@agnivade this does not work apparently:

~/Source/go/pkg/mod/cache/vcs/b44680b3c3708a854d4c89f55aedda0b223beb8d9e30fba969cefb5bd9c1e843$ git fetch --unshallow
remote: Counting objects: 20522, done
remote: Finding sources: 100% (19113/19113)
remote: Total 19113 (delta 12425), reused 18877 (delta 12425)
Receiving objects: 100% (19113/19113), 16.28 MiB | 5.61 MiB/s, done.
Resolving deltas: 100% (12425/12425), completed with 201 local objects.
error: Could not read a8576e2603b2bc62c97e830174b002dba60d6a0c
fatal: Failed to traverse parents of commit 44bcb96178d39510d2bb8a61ef1c08b85bb24a45
error: remote did not send all necessary objects
jefferai commented 5 years ago

Just hit this:

go: google.golang.org/api@v0.0.0-20181220000619-583d854617af: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /root/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: bad object 19e022d8cf43ce81f046bae8cc18c5397cc7732f
        error: remote did not send all necessary objects

This is with a totally clean modcache, as it's using a fresh Docker container for the build.

davidovich commented 5 years ago

We are having this error also @HazCod. Additionally, go fetch --unshallow in an unknown mod cache entry is not very actionable in a ci situation.

We are still having the error:

go: google.golang.org/api@v0.0.0-20181017004218-3f6e8463aa1d: git -c protocol.version=0 fetch --unshallow -f https://code.googlesource.com/google-api-go-client refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: bad object 19e022d8cf43ce81f046bae8cc18c5397cc7732f
        error: remote did not send all necessary objects

Is there an ETA to the upstream jgit fix ? I am having trouble making a repro as I don't know which of my transitive deps are wanting a version for google.golang.org/api@v0.0.0-20181017004218-3f6e8463aa1d, which I assume is causing the --unshallow request.

bcmills commented 5 years ago

The JGit fix has already been deployed, so the fact that folks are still running into this suggests that that fix did not cover all of the underlying causes.

bradfitz commented 5 years ago

The JGit fix also looks like it has no test (in https://git.eclipse.org/r/#/c/140463/), unless its tests are stored in a separate repo or something.

jayconrod commented 5 years ago

I've spoken with the Gerrit team. It sounds like Friday's CL wasn't a complete fix. They are working on fixing the remaining broken part.

draftcode commented 5 years ago

The JGit fix also looks like it has no test (in https://git.eclipse.org/r/#/c/140463/), unless its tests are stored in a separate repo or something.

I complained the git-core team that JGit has no e2e test environment and JGit doesn't have any shallow clone client support, and I cannot write a regression test.

lenaten commented 5 years ago

that works for me:

$ cd /go/pkg/mod/cache/vcs/<https://code.googlesource.com/google-api-go-client hash>
$ git fetch --unshallow -f https://github.com/googleapis/google-api-go-client
draftcode commented 5 years ago

It seems @jayconrod found a reproducible case. I'm trying to debug JGit again.

draftcode commented 5 years ago

jayconrod's case can be fixed by https://git.eclipse.org/r/c/140618/. I'm not sure if this covers other people's cases.

bcmills commented 5 years ago

The Gerrit team says the fix is fully rolled out now. Please let us know if you're still seeing these symptoms.

davidovich commented 5 years ago

The fix works for our case thanks!

anasanzari commented 5 years ago

@bcmills Issue still happens with docker build. image: golang:1.12-alpine

andybons commented 5 years ago

@draftcode @jrn

tparker commented 5 years ago

Are their new reproduction steps for the remaining failure(s)?

Sorry that this has been a session of whack-a-mole. Shallow clones make the server operate on the truncated history that a client is holding, which is a cross-cutting concern involving many different code paths and exposing a lot of corner cases.

If we can get clients to use partial clones (https://www.git-scm.com/docs/partial-clone) instead of shallow clones, where the partial clones pull down all history and tree objects, we are back to much more well worn and tested paths on the server.

dhui commented 5 years ago

Are their new reproduction steps for the remaining failure(s)?

$ git clone https://github.com/golang-migrate/migrate.git
$ cd migrate
$ docker build .

I haven't isolated the issue to a smaller reproducible step, but this is a starting point. The Dockerfile also uses the Alpine Linux image variant for Go 1.12.

bradfitz commented 5 years ago

I suspect Docker build layer caching is biting you. Disable the cache once and try again.

dhui commented 5 years ago

I'm using the latest Alpine Linux variant for Go 1.12. It doesn't seem like a docker build cache issue as the build also fails in TravisCI. I've included the command output with docker build --no-cache below. The error is near the bottom.

$ git clone https://github.com/golang-migrate/migrate.git
Cloning into 'migrate'...
remote: Enumerating objects: 3942, done.
remote: Total 3942 (delta 0), reused 0 (delta 0), pack-reused 3942
Receiving objects: 100% (3942/3942), 905.78 KiB | 2.41 MiB/s, done.
Resolving deltas: 100% (2334/2334), done.
$ cd migrate/
$ docker pull golang:1.12-alpine3.9
1.12-alpine3.9: Pulling from library/golang
Digest: sha256:1a5f8b6db670a7776ce5beeb69054a7cf7047a5d83176d39b94665a54cfb9756
Status: Image is up to date for golang:1.12-alpine3.9
$ docker build --no-cache .
Sending build context to Docker daemon  1.734MB
Step 1/14 : FROM golang:1.12-alpine3.9 AS downloader
 ---> b97a72b8e97d
Step 2/14 : ARG VERSION
 ---> Running in 1857cd80fdad
Removing intermediate container 1857cd80fdad
 ---> 510ee41d9f57
Step 3/14 : RUN apk add --no-cache git gcc musl-dev
 ---> Running in aa47db4f1b8b
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/17) Installing binutils (2.31.1-r2)
(2/17) Installing gmp (6.1.2-r1)
(3/17) Installing isl (0.18-r0)
(4/17) Installing libgomp (8.3.0-r0)
(5/17) Installing libatomic (8.3.0-r0)
(6/17) Installing libgcc (8.3.0-r0)
(7/17) Installing mpfr3 (3.1.5-r1)
(8/17) Installing mpc1 (1.0.3-r1)
(9/17) Installing libstdc++ (8.3.0-r0)
(10/17) Installing gcc (8.3.0-r0)
(11/17) Installing nghttp2-libs (1.35.1-r0)
(12/17) Installing libssh2 (1.8.2-r0)
(13/17) Installing libcurl (7.64.0-r1)
(14/17) Installing expat (2.2.6-r0)
(15/17) Installing pcre2 (10.32-r1)
(16/17) Installing git (2.20.1-r0)
(17/17) Installing musl-dev (1.1.20-r4)
Executing busybox-1.29.3-r10.trigger
OK: 117 MiB in 32 packages
Removing intermediate container aa47db4f1b8b
 ---> 490278082609
Step 4/14 : WORKDIR /go/src/github.com/golang-migrate/migrate
 ---> Running in 6454995f1a8a
Removing intermediate container 6454995f1a8a
 ---> 80bd63f3e551
Step 5/14 : COPY . ./
 ---> 85f649bc9ab3
Step 6/14 : ENV GO111MODULE=on
 ---> Running in 810b0af28aa2
Removing intermediate container 810b0af28aa2
 ---> f3039f731535
Step 7/14 : ENV DATABASES="postgres mysql redshift cassandra spanner cockroachdb clickhouse mongodb"
 ---> Running in fa10a30d7b20
Removing intermediate container fa10a30d7b20
 ---> aab8fb012995
Step 8/14 : ENV SOURCES="file go_bindata github aws_s3 google_cloud_storage gitlab"
 ---> Running in ddf8565db04d
Removing intermediate container ddf8565db04d
 ---> bb429ef15e9d
Step 9/14 : RUN go build -a -o build/migrate.linux-386 -ldflags="-X main.Version=${VERSION}" -tags "$DATABASES $SOURCES" ./cmd/migrate
 ---> Running in 24b16a1443fc
go: finding github.com/bitly/go-hostpool v0.0.0-20171023180738-a3a6125de932
go: finding github.com/gocql/gocql v0.0.0-20190301043612-f6df8288f9b4
go: finding github.com/xdg/stringprep v1.0.0
go: finding github.com/nakagami/firebirdsql v0.0.0-20190310045651-3c02a58cfed8
go: finding github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24
go: finding github.com/cockroachdb/apd v1.1.0
go: finding github.com/golang/protobuf v1.3.1
go: finding github.com/pkg/errors v0.8.1
go: finding google.golang.org/api v0.3.2
go: finding google.golang.org/grpc v1.19.1
go: finding github.com/fsouza/fake-gcs-server v1.5.0
go: finding github.com/konsorten/go-windows-terminal-sequences v1.0.2
go: finding github.com/aws/aws-sdk-go v1.17.7
go: finding github.com/tidwall/pretty v0.0.0-20180105212114-65a9db5fad51
go: finding github.com/dhui/dktest v0.3.0
go: finding google.golang.org/appengine v1.5.0
go: finding github.com/gogo/protobuf v1.2.1
go: finding golang.org/x/sys v0.0.0-20190405154228-4b34438f7a67
go: finding github.com/go-sql-driver/mysql v1.4.1
go: finding google.golang.org/appengine v1.1.0
go: finding golang.org/x/crypto v0.0.0-20190404164418-38d8ce5564a5
go: finding github.com/google/go-cmp v0.2.0
go: finding github.com/stretchr/testify v1.2.2
go: finding github.com/sirupsen/logrus v1.2.0
go: finding golang.org/x/net v0.0.0-20181220203305-927f97764cc3
go: finding github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0
go: finding golang.org/x/net v0.0.0-20180724234803-3673e40ba225
go: finding golang.org/x/tools v0.0.0-20190408220357-e5b8258f4918
go: finding cloud.google.com/go v0.36.0
go: finding github.com/hashicorp/golang-lru v0.5.1
go: finding gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
go: finding github.com/Microsoft/go-winio v0.4.11
go: finding github.com/stretchr/objx v0.1.1
go: finding cloud.google.com/go v0.26.0
go: finding github.com/stretchr/testify v1.3.0
go: finding golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3
go: finding github.com/satori/go.uuid v1.2.0
go: finding golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e
go: finding github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78
go: finding github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: finding github.com/golang/snappy v0.0.0-20170215233205-553a64147049
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/lib/pq v1.0.0
go: finding github.com/konsorten/go-windows-terminal-sequences v1.0.1
go: finding google.golang.org/grpc v1.19.0
go: finding github.com/gorilla/mux v1.7.1
go: finding github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c
go: finding github.com/google/go-github v17.0.0+incompatible
go: finding google.golang.org/api v0.1.0
go: finding github.com/docker/go-connections v0.4.0
go: finding golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
go: finding github.com/mattn/go-sqlite3 v1.10.0
go: finding google.golang.org/appengine v1.4.0
go: finding github.com/kr/pretty v0.1.0
go: finding google.golang.org/grpc v1.17.0
go: finding golang.org/x/tools v0.0.0-20190114222345-bf090417da8b
go: finding golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33
go: finding github.com/BurntSushi/toml v0.3.1
go: finding golang.org/x/net v0.0.0-20180826012351-8a410e7b638d
go: finding cloud.google.com/go v0.37.4
go: finding golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
go: finding honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099
go: finding golang.org/x/text v0.3.0
go: finding golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f
go: finding github.com/kshvakov/clickhouse v1.3.5
go: finding github.com/gorilla/mux v1.6.2
go: finding github.com/docker/distribution v2.7.0+incompatible
go: finding google.golang.org/genproto v0.0.0-20190404172233-64821d5d2107
go: finding golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go: finding golang.org/x/time v0.0.0-20181108054448-85acf8d2951c
go: finding github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed
go: finding golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be
go: finding gitlab.com/nyarla/go-crypt v0.0.0-20160106005555-d9a5dc2b789b
go: finding github.com/kisielk/gotool v1.0.0
go: finding github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5
go: finding github.com/sirupsen/logrus v1.4.1
go: finding github.com/gorilla/mux v1.7.0
go: finding golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3
go: finding github.com/kr/text v0.1.0
go: finding google.golang.org/api v0.3.1
go: finding github.com/jackc/fake v0.0.0-20150926172116-812a484cc733
go: finding golang.org/x/tools v0.0.0-20190312170243-e65039ee4138
go: finding github.com/golang/protobuf v1.2.0
go: finding golang.org/x/tools v0.0.0-20181030000716-a0a13e073c7b
go: finding honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a
go: finding github.com/docker/go-units v0.3.3
go: finding github.com/googleapis/gax-go/v2 v2.0.4
go: finding google.golang.org/genproto v0.0.0-20181202183823-bd91e49a0898
go: finding golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421
go: finding github.com/kr/pty v1.1.1
go: finding gotest.tools v2.2.0+incompatible
go: finding github.com/googleapis/gax-go/v2 v2.0.3
go: finding golang.org/x/oauth2 v0.0.0-20190111185915-36a7019397c4
go: finding github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57
go: finding google.golang.org/appengine v1.3.0
go: finding golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890
go: finding golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52
go: finding golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: finding github.com/hashicorp/go-multierror v1.0.0
go: finding github.com/davecgh/go-spew v1.1.0
go: finding golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2
go: finding github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c
go: finding go.opencensus.io v0.18.0
go: finding github.com/mongodb/mongo-go-driver v0.3.0
go: finding github.com/golang/mock v1.2.0
go: finding github.com/gogo/protobuf v1.2.0
go: finding github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
go: finding github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024
go: finding golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/hashicorp/errwrap v1.0.0
go: finding google.golang.org/genproto v0.0.0-20180831171423-11092d34479b
go: finding github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273
go: finding github.com/client9/misspell v0.3.4
go: finding github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding github.com/docker/docker v0.7.3-0.20190103212154-2b7e084dc98b
go: finding google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf
go: finding github.com/prometheus/client_golang v0.8.0
go: finding github.com/docker/docker v0.7.3-0.20190108045446-77df18c24acf
go: finding git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999
go: finding google.golang.org/grpc v1.14.0
go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: finding google.golang.org/grpc v1.16.0
go: finding github.com/cznic/ql v1.2.0
go: finding golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6
go: finding github.com/grpc-ecosystem/grpc-gateway v1.5.0
go: finding github.com/golang/snappy v0.0.1
go: finding github.com/opencontainers/image-spec v1.0.1
go: finding github.com/edsrzf/mmap-go v0.0.0-20170320065105-0bce6a688712
go: finding github.com/stretchr/objx v0.1.0
go: finding github.com/pkg/errors v0.8.0
go: finding github.com/cznic/lldb v1.1.0
go: finding golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961
go: finding github.com/openzipkin/zipkin-go v0.1.1
go: finding golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a
go: finding github.com/golang/lint v0.0.0-20180702182130-06c8688daad7
go: finding github.com/cznic/b v0.0.0-20180115125044-35e9bbe41f07
go: finding github.com/opencontainers/go-digest v1.0.0-rc1
go: finding github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
go: finding golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e
go: finding golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d
go: finding golang.org/x/sys v0.0.0-20180830151530-49385e6e1522
go: finding github.com/gorilla/context v1.1.1
go: finding golang.org/x/exp v0.0.0-20190121172915-509febef88a4
go: finding github.com/kisielk/errcheck v1.1.0
go: finding github.com/cznic/zappy v0.0.0-20160723133515-2533cb5b45cc
go: finding gopkg.in/inf.v0 v0.9.1
go: finding github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869
go: finding golang.org/x/tools v0.0.0-20180221164845-07fd8470d635
go: finding github.com/ghodss/yaml v1.0.0
go: finding github.com/xanzy/go-gitlab v0.15.0
go: finding github.com/golang/mock v1.1.1
go: finding golang.org/x/lint v0.0.0-20180702182130-06c8688daad7
go: finding cloud.google.com/go v0.34.0
go: finding github.com/cznic/internal v0.0.0-20180608152220-f44710a21d00
go: finding github.com/cockroachdb/cockroach-go v0.0.0-20181001143604-e0a95dfd547c
go: finding github.com/cznic/strutil v0.0.0-20171016134553-529a34b1c186
go: finding golang.org/x/net v0.0.0-20190213061140-3a22650c66bd
go: finding golang.org/x/sys v0.0.0-20190102155601-82a175fd1598
go: finding google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8
go: finding gopkg.in/yaml.v2 v2.2.1
go: finding github.com/cznic/mathutil v0.0.0-20180504122225-ca4c9f2c1369
go: finding golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288
go: finding golang.org/x/net v0.0.0-20181106065722-10aee1819953
go: finding github.com/cznic/golex v0.0.0-20170803123110-4ab7c5e190e4
go: finding github.com/jackc/pgx v3.2.0+incompatible
go: finding github.com/cznic/fileutil v0.0.0-20180108211300-6a051e75936f
go: finding golang.org/x/sys v0.0.0-20181029174526-d69651ed3497
go: finding github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c
go: finding github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
go: finding go.opencensus.io v0.20.1
go: finding github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9
go: finding github.com/google/martian v2.1.0+incompatible
go: finding github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82
go: finding dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c
go: finding github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
go: finding golang.org/x/net v0.0.0-20190311183353-d8887717615a
go: finding dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412
go: finding github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e
go: finding golang.org/x/tools v0.0.0-20190226205152-f727befe758c
go: finding github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537
go: finding github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc
go: finding golang.org/x/sync v0.0.0-20181108010431-42b317875d0f
go: finding github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9
go: finding github.com/apache/thrift v0.12.0
go: finding honnef.co/go/tools v0.0.0-20180728063816-88497007e858
go: finding dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3
go: finding github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
go: finding github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470
go: finding google.golang.org/genproto v0.0.0-20181029155118-b69ba1387ce2
go: finding github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625
go: finding github.com/cznic/sortutil v0.0.0-20150617083342-4c7342852e65
go: finding github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2
go: finding sourcegraph.com/sourcegraph/go-diff v0.5.0
go: finding github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab
go: finding github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191
go: finding github.com/openzipkin/zipkin-go v0.1.6
go: finding github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95
go: finding golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2
go: finding github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133
go: finding github.com/eapache/go-resiliency v1.1.0
go: finding github.com/gogo/protobuf v1.1.1
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding github.com/hashicorp/golang-lru v0.5.0
go: finding google.golang.org/api v0.0.0-20181030000543-1d582fd0359e
go: finding cloud.google.com/go v0.31.0
go: finding github.com/onsi/gomega v1.4.3
go: finding github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86
go: finding github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829
go: finding github.com/onsi/ginkgo v1.7.0
go: finding github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122
go: finding github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48
go: finding golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16
go: finding github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07
go: finding github.com/googleapis/gax-go v2.0.0+incompatible
go: finding github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241
go: finding go4.org v0.0.0-20180809161055-417644f6feb5
go: finding github.com/pierrec/lz4 v2.0.5+incompatible
go: finding github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f
go: finding github.com/Shopify/sarama v1.19.0
go: finding github.com/dustin/go-humanize v1.0.0
go: finding github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4
go: finding golang.org/x/net v0.0.0-20181029044818-c44066c5c816
go: finding google.golang.org/appengine v1.2.0
go: finding github.com/microcosm-cc/bluemonday v1.0.1
go: finding github.com/kr/pty v1.1.3
go: finding golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
go: finding github.com/google/go-querystring v1.0.0
go: finding github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
go: finding gopkg.in/fsnotify.v1 v1.4.7
go: finding github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
go: finding github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041
go: finding github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
go: finding github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50
go: finding github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239
go: finding github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e
go: finding google.golang.org/genproto v0.0.0-20190201180003-4b09977fb922
go: finding github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d
go: finding github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371
go: finding github.com/hpcloud/tail v1.0.0
go: finding golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a
go: finding github.com/fsnotify/fsnotify v1.4.7
go: finding dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0
go: finding golang.org/x/net v0.0.0-20181114220301-adae6a3d119a
go: finding github.com/prometheus/common v0.2.0
go: finding github.com/eapache/queue v1.1.0
go: finding golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5
go: finding github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20
go: finding github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc
go: finding github.com/Shopify/toxiproxy v2.1.4+incompatible
go: finding golang.org/x/net v0.0.0-20180906233101-161cd47e91fd
go: finding github.com/go-kit/kit v0.8.0
go: finding github.com/onsi/ginkgo v1.6.0
go: finding grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
go: finding github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
go: finding gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: finding gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: finding github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a
go: finding github.com/prometheus/client_golang v0.9.1
go: finding golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b
go: finding github.com/gliderlabs/ssh v0.1.1
go: finding github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf
go: finding github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d
go: finding github.com/sergi/go-diff v1.0.0
go: finding github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d
go: finding github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1
go: finding github.com/julienschmidt/httprouter v1.2.0
go: finding github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1
go: finding golang.org/x/net v0.0.0-20190125091013-d26f9f9a57f3
go: finding github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21
go: finding golang.org/x/net v0.0.0-20181108082009-03003ca0c849
go: finding github.com/russross/blackfriday v1.5.2
go: finding github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223
go: finding golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
go: finding google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19
go: finding golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4
go: finding github.com/go-stack/stack v1.8.0
go: finding sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4
go: finding github.com/go-logfmt/logfmt v0.3.0
go: google.golang.org/api@v0.0.0-20180910000450-7ca32eb868bf: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
    fatal: missing tree object 'dcbf3e3c23816703a8f5ebfe6583cc3fc2793526'
    error: remote did not send all necessary objects
go: error loading module requirements
The command '/bin/sh -c go build -a -o build/migrate.linux-386 -ldflags="-X main.Version=${VERSION}" -tags "$DATABASES $SOURCES" ./cmd/migrate' returned a non-zero code: 1
$

Maybe it's an issue w/ the alpine linux 3.9 version of git (v2.20.1-r0)? Local builds (on Mac OS X with git 2.21.0) succeed.

bradfitz commented 5 years ago

@dhui, that all worked for me, though. Does it always fail for you, or just sometimes?

dhui commented 5 years ago

Does it always fail for you, or just sometimes?

It's always failed for me. On TravisCI, it failed 2/2 times and when running locally in docker, it failed 5/5 times. The missing tree object varies between runs though.

bradfitz commented 5 years ago

Okay, on my second try it also fails for me.

go: google.golang.org/api@v0.0.0-20180910000450-7ca32eb868bf: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: missing tree object 'dcbf3e3c23816703a8f5ebfe6583cc3fc2793526'
        error: remote did not send all necessary objects
go: error loading module requirements

I'm reopening.

bcmills commented 5 years ago

@draftcode, is someone on the Gerrit side looking into this?

draftcode commented 5 years ago

Reproduced with JGit:

git init . --bare
git remote add origin git://localhost:9418/google-api-go-client.git
git fetch -f --depth=1 origin refs/tags/v0.3.2:refs/tags/v0.3.2
git fetch -f --depth=1 origin refs/tags/v0.1.0:refs/tags/v0.1.0
git fetch -f --depth=1 origin refs/tags/v0.3.1:refs/tags/v0.3.1
git fetch -f --unshallow origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
draftcode commented 5 years ago

I found another bug in JGit code that made it exclude dcbf3e3c23816703a8f5ebfe6583cc3fc2793526. But after I fix that, I found another bug that doesn't send a parent of v0.1.0. I'm debugging JGit more.

draftcode commented 5 years ago

https://git.eclipse.org/r/c/140900/ and https://git.eclipse.org/r/c/140901/ will fix the the three tag unshallow case. Tried all three previously reproduced cases.

iamtakingiteasy commented 5 years ago

A possible temporary workaround for docker builds might be

RUN ( \
      export GO111MODULE=on && cd / && go get google.golang.org/api && \
      cd "$(grep -l https://code.googlesource.com/google-api-go-client $GOPATH/pkg/mod/cache/vcs/*.info | sed 's/\.info$//')" && \
      git fetch -f https://code.googlesource.com/google-api-go-client 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*' \
    )
jayconrod commented 5 years ago

Still seems to be happening. Pinged internal bug.

draftcode commented 5 years ago

https://github.com/golang/go/issues/31399#issuecomment-485039238 is not rolled out yet.

jayconrod commented 5 years ago

Oops, I thought those CLs had been deployed already.

@draftcode Do you expect this will fix the GOPATH=/tmp/go GO111MODULE=on go install go.chromium.org/goma/server/cmd/remoteexec_proxy command from the original example?

anasanzari commented 5 years ago

@iamtakingiteasy not working anymore

iamtakingiteasy commented 5 years ago

@anasanzari true, but it seems original issue of fetching https://code.googlesource.com/google-api-go-client from go mod tools was somehow fixed despite commits not being merged (or at least I couldn't find a way to reproduce), this workaround is not needed anymore.

You can remove it from your pipeline and build as normal if that was your use-case.

Also it is indeed odd such fetch would not work. Might be somewhat related issue (which those commits might fix).

jayconrod commented 5 years ago

All fixes are rolled out now, and this seems to be working. Thanks @draftcode for working on this!

I'll close this but everyone please comment if you're still seeing this, and we'll investigate.

anasanzari commented 5 years ago

@jayconrod issue is still there. I've added my case in https://github.com/anasanzari/golang_build_check. docker build . fails with the same error.