gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.25k stars 1.73k forks source link

Investigate spurious package download errors when running in CI #6354

Closed webvictim closed 3 years ago

webvictim commented 3 years ago

I'm seeing a lot of Go module downloads being attempted when running lint/test targets in CI. I'm pretty sure this shouldn't be happening on the main Teleport codebase as all dependencies should be vendored. Investigate why this is happening and fix it.

This may be related to the api directory (as I think we intend that dependencies should be downloaded for that package specifically) and if so, we should modify CI to write the cache to a writable temp dir to remove the errors.

docker run --rm=true -v "$(pwd)/../":/go/src/github.com/gravitational/teleport -v /tmp:/tmp -w /go/src/github.com/gravitational/teleport -h buildbox -v /tmpfs/go/kubeconfig.ci:/mnt/kube/config -e KUBECONFIG=/mnt/kube/config -e TEST_KUBE=true -v /tmpfs/go/cache:/go/cache -e GOCACHE=/go/cache -v "$(pwd)/bcc:/usr/include/bcc" -u 1000:1000 -t quay.io/gravitational/teleport-buildbox:go1.16.2 \
    /bin/bash -c "make -C /go/src/github.com/gravitational/teleport FLAGS='-cover' integration"
make: Entering directory '/go/src/github.com/gravitational/teleport'
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/jonboulle: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/gopkg.in: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/niemeyer: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/google: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/gogo: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/sirupsen: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/stretchr: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/gravitational: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/golang: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/net/@v/v0.0.0-20190311183353-d8887717615a.mod496193015.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/konsorten: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/golang.org/x/oauth2: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/golang: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/gopkg.in: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/net/@v/v0.0.0-20200707034311-ab3426394381.mod277341737.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/google: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/gopkg.in: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/kr: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/golang.org/x/lint: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/tools/@v/v0.0.0-20190226205152-f727befe758c.mod168565194.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/envoyproxy: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/golang: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/davecgh: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/kr: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/github.com/kisielk/errcheck/@v/v1.2.0.mod890951957.tmp: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/sys/@v/v0.0.0-20190422165155-953cdadca894.mod920627189.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/prometheus: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/client9: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/honnef.co: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/golang: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/golang.org/x/exp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/cloud.google.com: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/golang: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/sys/@v/v0.0.0-20200323222414-85ca7c5b95cd.mod568160156.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/!burnt!sushi: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/net/@v/v0.0.0-20190213061140-3a22650c66bd.mod926275561.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/pmezard: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/sync/@v/v0.0.0-20181108010431-42b317875d0f.mod980435746.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/envoyproxy: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/golang.org/x/lint: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/tools/@v/v0.0.0-20181030221726-6c7e314b6563.mod990689002.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/census-instrumentation: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/sync/@v/v0.0.0-20180314180146-1d60e4601c6f.mod526661577.tmp: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/sys/@v/v0.0.0-20180830151530-49385e6e1522.mod210707463.tmp: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/tools/@v/v0.0.0-20190114222345-bf090417da8b.mod394767996.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/golang: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/net/@v/v0.0.0-20180826012351-8a410e7b638d.mod623118623.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/golang.org/x/lint: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/tools/@v/v0.0.0-20190524140312-2c0ae7006135.mod924879241.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/honnef.co: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/net/@v/v0.0.0-20180724234803-3673e40ba225.mod548653033.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: open /go/pkg/mod/cache/download/golang.org/x/tools/@v/v0.0.0-20190311212946-11955173bddd.mod183653891.tmp: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/golang: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/golang: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/golang: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/golang: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/google: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/google.golang.org: permission denied
go: writing go.mod cache: mkdir /go/pkg/mod/cache/download/github.com/google: permission denied
go: downloading github.com/gravitational/trace v1.1.13
go: downloading github.com/gogo/protobuf v1.3.1
go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go: downloading github.com/golang/protobuf v1.4.2
go: downloading google.golang.org/grpc v1.27.0
go: downloading golang.org/x/net v0.0.0-20200707034311-ab3426394381
go: downloading gopkg.in/yaml.v2 v2.4.0
KUBECONFIG is: /mnt/kube/config, TEST_KUBE: true
go test -tags "pam  bpf" github.com/gravitational/teleport/e/integration github.com/gravitational/teleport/integration -cover
ok      github.com/gravitational/teleport/e/integration 0.061s  coverage: [no statements]\
webvictim commented 3 years ago

This seems to be related to all go related commands running inside Docker - make -C build.assets release, make -C build.assets lint etc are all affected. It also seems to be related to Go 1.16 as it doesn't happen on the 1.15.5 buildbox.

russjones commented 3 years ago

@webvictim Still seeing this issue on master.

$ make -C build.assets release
make: Entering directory '/home/rjones/go/src/github.com/gravitational/teleport/build.assets'
grep: /Makefile: No such file or directory
if [[ "quay.io/gravitational/teleport-buildbox:go1.16.2" == "quay.io/gravitational/teleport-buildbox:go1.16.2" ]]; then \
    if [[ ${DRONE} == "true" ]] && ! docker inspect --type=image quay.io/gravitational/teleport-buildbox:go1.16.2 2>&1 >/dev/null; then docker pull quay.io/gravitational/teleport-buildbox:go1.16.2 || true; fi; \
    docker build \
        --build-arg UID=$(id -u) \
        --build-arg GID=$(id -g) \
        --build-arg RUNTIME=go1.16.2 \
        --build-arg PROTOC_VER=3.6.1 \
        --build-arg GOGO_PROTO_TAG=v1.3.2 \
        --build-arg PROTOC_PLATFORM=linux-x86_64 \
        --cache-from quay.io/gravitational/teleport-buildbox:go1.16.2 \
        --tag quay.io/gravitational/teleport-buildbox:go1.16.2 . ; \
fi
/bin/sh: 1: [[: not found
docker run --rm=true -v "$(pwd)/../":/go/src/github.com/gravitational/teleport -v /tmp:/tmp -w /go/src/github.com/gravitational/teleport -h buildbox -e GOMODCACHE=/tmp/gomodcache -v "$(pwd)/bcc:/usr/include/bcc" -i -u $(id -u):$(id -g) quay.io/gravitational/teleport-buildbox:go1.16.2 \
    /usr/bin/make release -e ADDFLAGS="" OS=linux ARCH=amd64 RUNTIME=go1.16.2
Unable to find image 'quay.io/gravitational/teleport-buildbox:go1.16.2' locally
go1.16.2: Pulling from gravitational/teleport-buildbox
4bbfd2c87b75: Pull complete 
d2e110be24e1: Pull complete 
889a7173dcfe: Pull complete 
4477964855da: Pull complete 
d1cd544bb33a: Pull complete 
a976449758c8: Pull complete 
56d5d9787fa2: Pull complete 
6000c3182e63: Pull complete 
dbc3e5741256: Pull complete 
c5ca5ec0ffd9: Pull complete 
da8e6f435132: Pull complete 
a778b354a73b: Pull complete 
c9daad76e250: Pull complete 
2081a2c72b7f: Pull complete 
89f995e604e4: Pull complete 
6ed1a2c55935: Pull complete 
Digest: sha256:9f541bd755637fd36b36e4033aeb7cf62ad35a6ff7d7b35140fd5de581a0abe7
Status: Downloaded newer image for quay.io/gravitational/teleport-buildbox:go1.16.2
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
---> Building with GOOS=linux GOARCH=amd64 and with PAM support and without FIPS support and with BPF support.
/usr/bin/make --no-print-directory release-unix
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
---> Cleaning up OSS build artifacts.
rm -rf build
go clean -cache
rm -rf `go env GOPATH`/pkg/`go env GOHOSTOS`_`go env GOARCH`/github.com/gravitational/teleport*
rm -rf teleport
rm -rf *.gz
rm -rf *.zip
rm -f gitref.go
---> Building OSS web assets.
/usr/bin/make all WEBASSETS_TAG="webassets_embed"
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
VERSION=7.0.0-dev /usr/bin/make -f version.mk setver
---> Building OSS binaries.
/usr/bin/make build/teleport build/tctl build/tsh
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go build -tags "pam  bpf webassets_embed" -o build/teleport  -ldflags '-w -s' ./tool/teleport
failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
Makefile:124: recipe for target 'build/teleport' failed
make[3]: *** [build/teleport] Error 1
make[2]: *** [all] Error 2
Makefile:110: recipe for target 'all' failed
make[1]: *** [full] Error 2
Makefile:138: recipe for target 'full' failed
Makefile:172: recipe for target 'release' failed
make: *** [release] Error 2
make: *** [Makefile:247: release] Error 2
make: Leaving directory '/home/rjones/go/src/github.com/gravitational/teleport/build.assets'
webvictim commented 3 years ago

This is a different error. Having recently switched to MacOS I think it might be related to Docker on that platform as I’ve seen this message too. I’ll take a look.

webvictim commented 3 years ago

@russjones I can't repro this using current master (eb7bb01d34a84a253e7a46f51ac53d17c76c8654) on either MacOS or Linux.

The error looks like it could be related to $HOME being overridden to / in Docker. Can you please run go env GOCACHE inside a build container and show the output?

Also, what version of Docker are you running? The output of docker info would be helpful.

webvictim commented 3 years ago

Closing this due to lack of repro. The original issue was fixed in #6899. Please raise a new issue if this behaviour continues.