cmd/go: git remote (Gerrit) did not send all necessary objects

ukai commented 5 years ago

What version of Go are you using (`go version`)?

$ go version
go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/ukai/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr/local/google/home/ukai/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/google-golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build741696436=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ ls /tmp/go
ls: cannot access '/tmp/go': No such file or directory
$ mkdir /tmp/go
$ GOPATH=/tmp/go GO111MODULE=on go install go.chromium.org/goma/server/cmd/remoteexec_proxy

What did you expect to see?

go modules downloaded and binary is built.

What did you see instead?

...
go: finding grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
go: google.golang.org/api@v0.0.0-20181030000543-1d582fd0359e: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: bad object c7964aab01c00a07d0cc6216a78a5570459cebfd
        error: remote did not send all necessary objects
..
go: error loading module requirements

I could fix it by

$ cd /tmp/go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0; git fetch --unshallow

then go install works as expected.

https://bugs.chromium.org/p/chromium/issues/detail?id=951546

bradfitz commented 5 years ago

@anasanzari, thanks. I can reproduce the failure.

Gerrit/jgit is still broken:

go: google.golang.org/api@v0.0.0-20181030000543-1d582fd0359e: git -c protocol.version=0 fetch --unshallow -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/9e62a95b0409d58bc0130bae299bdffbc7b7e74f3abe1ecf897474cc474b8bc0: exit status 128:
        fatal: missing tree object 'dcbf3e3c23816703a8f5ebfe6583cc3fc2793526'
        error: remote did not send all necessary objects
go: error loading module requirements

Reopening.

bradfitz commented 5 years ago

@draftcode, please investigate?

draftcode commented 5 years ago

I can reproduce this, but I cannot reproduce this with JGit.

anasanzari commented 5 years ago

can someone give a quick fix for this?

tparker commented 5 years ago

The reproduction case above only failed in production-like environments, so it took longer to track down. The next fix forgoes potential data transfer optimizations in the name of correctness. Unshallow operations will be full clones. We will work on reintroducing the optimizations but that is a multi-week project that won't begin immediately.

The patch is in our release process, building and verifying. It should start rolling it out in a few hours.

tparker commented 5 years ago

Sorry to say we hit a snag in the release process. We are working through it and will have a release ready later this afternoon. But releasing on a Friday afternoon is...dicey. We will wait until Monday to update the servers.

Good news is that the fix bypasses all the finicky calculations that have caused this problem to recur, and we have verified it in a production-like environment.

tparker commented 5 years ago

This was fully deployed by yesterday afternoon, PDT.

jayconrod commented 5 years ago

Thanks @tparker and @draftcode for working on this. I'll close this issue now.

If anyone is still seeing this, please let us know.

anjmao commented 5 years ago

@jayconrod Hi, I still see this issue:

go get -y github.com/jinzhu/gorm

I get error:

go: golang.org/x/net@v0.0.0-20190125091013-d26f9f9a57f3: git fetch -f https://go.googlesource.com/net refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/anjmao/go/pkg/mod/cache/vcs/4a22365141bc4eea5d5ac4a1395e653f2669485db75ef119e7bbec8e19b12a21: exit status 1:
    error: Could not read 65e2d4e15006aab9813ff8769e768bbf4bb667a0
    fatal: bad object 65e2d4e15006aab9813ff8769e768bbf4bb667a0
    error: https://go.googlesource.com/net did not send all necessary objects

tparker commented 5 years ago

Does anyone have a full reproduction recipe? I'm unable to reproduce it.

jayconrod commented 5 years ago

@anjmao I couldn't reproduce this. Are you seeing this consistently? Does this happen with an empty module cache (i.e., export GOPATH=$(mktemp -d))? Anything unusual about your environment (git config, git version, proxy settings)?

If you can give us a sequence of git commands that reproduces this (i.e., from go get -x), that would be very helpful.

golang / go