Closed travisgroth closed 3 years ago
travisgroth
commented
4 years ago 🤔 It looks like in https://github.com/travisgroth/renovate-test-tidy, tidy is actually being run, which is strange. The postUpdateOptions is copied from https://github.com/pomerium/pomerium, where it does not seem to be running.
Edit for clarity: not true. The change is more complete but not as if tidy ran.
rarkins
commented
4 years ago We literally run the tidy twice if it's configured, in an attempt to squash this bug. I am still not sure what's causing the problem. FYI the app runs with binaryMode=docker, which may be a contributor. Even when you see a failing PR, you should still see the tidy being run in the logs.
I'm a bit confused by your links though, can you help me understand?
https://github.com/pomerium/pomerium/pull/1381/files is a merged PR, created by the app. It has a single commit, seems to have the correct line there. Did it work as expected, or did you force push to fix it?
The debug logs you produce show the following command being run: go get -d ./... && go mod tidy && go mod tidy. You said "it does not seem to be running" though - can you confirm?
Meanwhile https://github.com/travisgroth/renovate-test/pull/2/files is your attempt to reproduce the issue? Is there a problem line there, e.g. the 1.7.0 should be removed?
travisgroth
commented
4 years ago We literally run the tidy twice if it's configured, in an attempt to squash this bug. I am still not sure what's causing the problem. FYI the app runs with binaryMode=docker, which may be a contributor. Even when you see a failing PR, you should still see the tidy being run in the logs.
Yes I can see that it should be run that way in the debug logs but the results just don't match. I'm honestly not even sure how to arrive at the state we get. Internally, do you manually manipulate the go.mod and then let go get -d handle the rest? It's weird that the results seem to have changed over time, or are inconsistent.
https://github.com/pomerium/pomerium/pull/1381/files is a merged PR, created by the app. It has a single commit, seems to have the correct line there. Did it work as expected, or did you force push to fix it?
The app-created PR is missing a line. The diff should have had 3 lines and looked like this:
--- a/go.mod
+++ b/go.mod
@@ -57,7 +57,7 @@ require (
github.com/stretchr/testify v1.6.1
github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80
go.opencensus.io v0.22.4
- go.uber.org/zap v1.15.0
+ go.uber.org/zap v1.16.0
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
golang.org/x/net v0.0.0-20200904194848-62affa334b73
golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43
diff --git a/go.sum b/go.sum
index 5194c126..42ad65a7 100644
--- a/go.sum
+++ b/go.sum
@@ -524,6 +524,8 @@ go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9E
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
go.uber.org/zap v1.15.0 h1:ZZCA22JRF2gQE5FoNmhmrf7jeJJ2uhqDUNRYKm8dvmM=
go.uber.org/zap v1.15.0/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc=
+go.uber.org/zap v1.16.0 h1:uFRZXykJGK9lLY4HtgSw44DnIcAM+kRBP7x5m+NpAOM=
+go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ=We fixed it with a manual follow up here https://github.com/pomerium/pomerium/pull/1384/files. All that was run was go mod tidy. You can see the missing line for zap (and several other deps) were added.
The debug logs you produce show the following command being run: go get -d ./... && go mod tidy && go mod tidy. You said "it does not seem to be running" though - can you confirm?
Is there a better way to get the last exec output for that branch? I pasted all the logs when filtered for that branch. If go mod tidy had been run on that branch, the PR would have had the 3 expected lines from the zap change, plus cleanup from the previous PRs that were missing lines. The results just don't match what happens when I run the underlying commands locally in that state.
Moving to the reproduction repos...I see two different issues. Neither is correct but I'm not sure how we get the results in the PRs, unless there is a different command sequence.
Without tidy (https://github.com/travisgroth/renovate-test), the renovate PR winds up with a missing line compared to running just go get -d ./... manually:
https://github.com/travisgroth/renovate-test/pull/2/files
vs
With tidy (https://github.com/travisgroth/renovate-test-tidy), I get the expected 3 expected lines added in the renovate PR, but the old lines in go.sum are not removed. Again, this differs from running the commands manually:
https://github.com/travisgroth/renovate-test-tidy/pull/2/files
vs
Meanwhile https://github.com/travisgroth/renovate-test/pull/2/files is your attempt to reproduce the issue? Is there a problem line there, e.g. the 1.7.0 should be removed?
See above. Without tidy, I wouldn't expect the 1.7.0 line to be removed but I would expect the 3 line change, not two.
Hope that helps. Happy to dig in further or try things as needed. Thanks!
rarkins
commented
4 years ago Internally, do you manually manipulate the go.mod and then let
go get -dhandle the rest?
Yes, that's right. But also note that unlike most other package managers, the go x commands can change not only the go.sum but also the go.mod at times so we check in case either file has been updated.
BTW we have found that the go x commands aren't perfect, e.g. go mod vendor can actually modify the go.* files too, so we run a tidy after it even though we run a tidy before it too. It's possible the problem is in go, or with us, or a bit of both.
What version of go are you running with locally? If you have docker installed, can you experiment with running our dockerized versions of go? e.g. something like in the logs:
docker run --rm -v `pwd`:`pwd` -w `pwd` renovate/golang go get -d ./..Adding some more logs from that particular branch/PR:
{"msg":"gomod.updateDependency: v0.0.0-20200822124328-c89045814202"}
{"depName":"golang.org/x/net","lineToChange":"\tgolang.org/x/net v0.0.0-20200822124328-c89045814202","newDigestRightSized":"62affa334b73","msg":"gomod: need to update digest"}
{"packageFile":"go.mod","depName":"golang.org/x/net","msg":"Updating packageFile content"}
{"msg":"gomod.updateArtifacts(go.mod)"}
{"msg":"Using GOPATH: /tmp/renovate-cache/others/go"}
{"cmd":"go","args":"get -d ./...","msg":"go get command included"}
{"cmd":"go","args":"mod tidy","msg":"go mod tidy command included"}
{"cmd":"go","args":"mod tidy","msg":"additional go mod tidy command included"}
{"msg":"Using docker to execute"}
{"constraint":"1.14","msg":"Found npm version constraint - checking for a compatible renovate/go image to use"}
{"msg":"getLabels(https://index.docker.io, renovate/go, latest)"}
{"msg":"Missing versioning"}
{"constraint":"1.14","version":"1.14.5","msg":"Found compatible npm version"}
{"image":"renovate/go","tagConstraint":"1.14","tagVersioning":"npm","tag":"1.14.5","msg":"Resolved tag constraint"}
{"msg":"Fetching Docker image: renovate/go:1.14.5"}
{"msg":"Finished fetching Docker image"}
{"command":"docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/pomerium/pomerium\":\"/mnt/renovate/gh/pomerium/pomerium\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/pomerium/pomerium\" renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://github.com/\\\" && go get -d ./... && go mod tidy && go mod tidy\"","msg":"Executing command"}
< see output below>
{"msg":"Returning updated go.sum"}
{"msg":"Found updated go.mod after go.sum update"}And then the output:
go: downloading github.com/gomodule/redigo v1.8.2
go: downloading github.com/gorilla/handlers v1.5.0
go: downloading contrib.go.opencensus.io/exporter/prometheus v0.2.0
go: downloading github.com/envoyproxy/go-control-plane v0.9.6
go: downloading go.uber.org/zap v1.15.0
go: downloading github.com/caddyserver/certmagic v0.11.3-0.20200826185716-09acc6bf59fa
go: downloading gopkg.in/ini.v1 v1.51.1
go: downloading github.com/klauspost/cpuid v1.2.5
go: downloading cloud.google.com/go v0.62.0
go: downloading github.com/libdns/libdns v0.1.0
go: downloading github.com/openzipkin/zipkin-go v0.2.3
go: downloading github.com/onsi/gocleanup v0.0.0-20140331211545-c1a5478700b5
go: downloading github.com/hashicorp/memberlist v0.2.2
go: downloading gopkg.in/cookieo9/resources-go.v2 v2.0.0-20150225115733-d27c04069d0d
go: downloading github.com/envoyproxy/protoc-gen-validate v0.1.0
go: downloading github.com/cenkalti/backoff/v4 v4.0.2
go: downloading github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354
go: downloading github.com/open-policy-agent/opa v0.23.2
go: downloading github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0
go: downloading github.com/hashicorp/go-sockaddr v1.0.0
go: downloading github.com/felixge/httpsnoop v1.0.1
go: downloading github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80
go: downloading contrib.go.opencensus.io/exporter/zipkin v0.1.2
go: downloading github.com/pomerium/csrf v1.6.2-0.20190918035251-f3318380bad3
go: downloading github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529
go: downloading github.com/btcsuite/btcutil v1.0.2
go: downloading github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da
go: downloading contrib.go.opencensus.io/exporter/jaeger v0.2.1
go: downloading github.com/rakyll/statik v0.1.7
go: downloading github.com/prometheus/statsd_exporter v0.15.0
go: downloading github.com/mholt/acmez v0.1.1-0.20200811184240-dc9c5f05ed1e
go: downloading github.com/hashicorp/go-msgpack v0.5.3
go: downloading github.com/lithammer/shortuuid/v3 v3.0.4
go: downloading github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966
go: downloading github.com/hashicorp/go-multierror v1.0.0
go: downloading github.com/miekg/dns v1.1.30
go: downloading github.com/google/go-jsonnet v0.16.0
go: downloading github.com/google/btree v1.0.0
go: downloading github.com/mitchellh/hashstructure v1.0.0
go: downloading github.com/gorilla/securecookie v1.1.1
go: downloading google.golang.org/genproto v0.0.0-20200829155447-2bf3329a0021
go: downloading github.com/census-instrumentation/opencensus-proto v0.2.1
go: downloading github.com/natefinch/atomic v0.0.0-20200526193002-18c0533a5b09
go: downloading github.com/pelletier/go-toml v1.6.0
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/yashtewari/glob-intersection v0.0.0-20180916065949-5c77d914dd0b
go: downloading github.com/OneOfOne/xxhash v1.2.7
go: downloading golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d
go: downloading github.com/golang/mock v1.4.4
go: downloading github.com/onsi/ginkgo v1.11.0
go: downloading github.com/nsf/jsondiff v0.0.0-20200515183724-f29ed568f4ce
go: downloading github.com/onsi/gomega v1.8.1
go: downloading github.com/hpcloud/tail v1.0.0
go: downloading github.com/lib/pq v0.0.0-20180327071824-d34b9ff171c2
go: downloading github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c
go: downloading github.com/cenkalti/backoff/v3 v3.0.0
go: downloading gotest.tools/v3 v3.0.2
go: downloading github.com/containerd/continuity v0.0.0-20190827140505-75bee3e2ccb6The above is captured all in stderr and stdout is empty - I'm not sure if that's expected or not. I don't see a confirmation that tidy is run, despite the CLI arguments. What if go get exits so go mod tidy is never called?
Any chance GOPATH or CGO_ENABLED cause an issue?
rarkins
commented
4 years ago I forked the main repo, rolled back the zap update, and then tried to reproduce locally. The results are confusing.
The first time it ran, it failed and printed an Artifact error and added that to the PR
go: github.com/openzipkin/zipkin-go@v0.2.3 requires\n\tgithub.com/eapache/queue@v1.1.0: reading github.com/eapache/queue/go.mod at revision v1.1.0: unknown revision v1.1.0The second time, it worked and removed the error comment, also updating the commit: https://github.com/renovate-tests/pomerium/pull/1/files
travisgroth
commented
4 years ago What version of go are you running with locally? If you have docker installed, can you experiment with running our dockerized versions of go? e.g. something like in the logs:
Yep, but I've been doing it with upstream golang images. 1.14.0, 1.14.5 (to match your version) and 1.15.1.
Adding some more logs from that particular branch/PR:
Here's an attempt to run the same setup as you have there. Intentionally leaving out tidy:
% docker run -it -v $(pwd):/mnt/renovate/gh/pomerium/pomerium -e GOPATH=/tmp/renovate-cache/others/go -e CGO_ENABLED=false -w "/mnt/renovate/gh/pomerium/pomerium" renovate/go:1.14.5 bash -l -c "git diff && go get -d ./..."
diff --git a/go.mod b/go.mod
index ce5e0062..abeba3a5 100644
--- a/go.mod
+++ b/go.mod
@@ -57,7 +57,7 @@ require (
github.com/stretchr/testify v1.6.1
github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80
go.opencensus.io v0.22.4
- go.uber.org/zap v1.15.0
+ go.uber.org/zap v1.16.0
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
golang.org/x/net v0.0.0-20200904194848-62affa334b73
golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43
go: downloading github.com/rs/cors v1.7.0
go: downloading github.com/caddyserver/certmagic v0.11.3-0.20200826185716-09acc6bf59fa
go: downloading github.com/coreos/go-oidc v2.2.1+incompatible
go: downloading github.com/golang/protobuf v1.4.2
go: downloading gopkg.in/square/go-jose.v2 v2.5.1
go: downloading github.com/hashicorp/memberlist v0.2.2
go: downloading github.com/fsnotify/fsnotify v1.4.9
go: downloading github.com/google/btree v1.0.0
go: downloading github.com/google/go-cmp v0.5.2
go: downloading github.com/pomerium/csrf v1.6.2-0.20190918035251-f3318380bad3
go: downloading github.com/libdns/libdns v0.1.0
go: downloading gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637
go: downloading google.golang.org/protobuf v1.25.0
go: downloading google.golang.org/grpc v1.31.1
go: downloading github.com/miekg/dns v1.1.30
go: downloading github.com/mitchellh/hashstructure v1.0.0
go: downloading github.com/prometheus/client_golang v1.7.1
go: downloading gopkg.in/yaml.v2 v2.3.0
go: downloading github.com/hashicorp/go-multierror v1.0.0
go: downloading go.opencensus.io v0.22.4
go: downloading github.com/rakyll/statik v0.1.7
go: downloading github.com/klauspost/cpuid v1.2.5
go: downloading github.com/rs/zerolog v1.19.0
go: downloading github.com/prometheus/procfs v0.1.3
go: downloading github.com/envoyproxy/go-control-plane v0.9.6
go: downloading github.com/hashicorp/errwrap v1.0.0
go: downloading github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80
go: downloading github.com/hashicorp/go-msgpack v0.5.3
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/openzipkin/zipkin-go v0.2.3
go: downloading github.com/prometheus/common v0.10.0
go: downloading contrib.go.opencensus.io/exporter/jaeger v0.2.1
go: downloading github.com/gorilla/mux v1.8.0
go: downloading github.com/beorn7/perks v1.0.1
go: downloading golang.org/x/net v0.0.0-20200904194848-62affa334b73
go: downloading github.com/spf13/cobra v1.0.0
go: downloading golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
go: downloading github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966
go: downloading github.com/gorilla/securecookie v1.1.1
go: downloading golang.org/x/sys v0.0.0-20200803210538-64077c9b5642
go: downloading github.com/uber/jaeger-client-go v2.25.0+incompatible
go: downloading github.com/cenkalti/backoff/v4 v4.0.2
go: downloading go.uber.org/zap v1.16.0
go: downloading github.com/btcsuite/btcutil v1.0.2
go: downloading github.com/mholt/acmez v0.1.1-0.20200811184240-dc9c5f05ed1e
go: downloading github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529
go: downloading gopkg.in/cookieo9/resources-go.v2 v2.0.0-20150225115733-d27c04069d0d
go: downloading go.uber.org/atomic v1.6.0
go: downloading github.com/spf13/viper v1.7.1
go: downloading google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d
go: downloading github.com/cespare/xxhash v1.1.0
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/spf13/afero v1.2.2
go: downloading github.com/cespare/xxhash/v2 v2.1.1
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading golang.org/x/text v0.3.3
go: downloading github.com/magiconair/properties v1.8.1
go: downloading github.com/stretchr/testify v1.6.1
go: downloading google.golang.org/api v0.30.0
go: downloading github.com/pelletier/go-toml v1.6.0
go: downloading github.com/spf13/jwalterweatherman v1.1.0
go: downloading github.com/envoyproxy/protoc-gen-validate v0.1.0
go: downloading github.com/pquerna/cachecontrol v0.0.0-20180517163645-1555304b9b35
go: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/census-instrumentation/opencensus-proto v0.2.1
go: downloading github.com/hashicorp/golang-lru v0.5.4
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/gomodule/redigo v1.8.2
go: downloading github.com/google/uuid v1.1.2
go: downloading github.com/subosito/gotenv v1.2.0
go: downloading gopkg.in/ini.v1 v1.51.1
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go: downloading contrib.go.opencensus.io/exporter/prometheus v0.2.0
go: downloading github.com/open-policy-agent/opa v0.23.2
go: downloading github.com/gorilla/handlers v1.5.0
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/prometheus/statsd_exporter v0.15.0
go: downloading github.com/google/go-jsonnet v0.16.0
go: downloading golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
go: downloading golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43
go: downloading github.com/hashicorp/go-sockaddr v1.0.0
go: downloading github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da
go: downloading go.uber.org/multierr v1.5.0
go: downloading github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354
go: downloading github.com/sirupsen/logrus v1.4.2
go: downloading contrib.go.opencensus.io/exporter/zipkin v0.1.2
go: downloading github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0
go: downloading cloud.google.com/go v0.65.0
go: downloading github.com/felixge/httpsnoop v1.0.1
go: downloading google.golang.org/appengine v1.6.6
go: downloading github.com/hashicorp/go-immutable-radix v1.0.0
go: downloading github.com/natefinch/atomic v0.0.0-20200526193002-18c0533a5b09
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.2
go: downloading github.com/lithammer/shortuuid/v3 v3.0.4
go: downloading github.com/mitchellh/mapstructure v1.1.2
go: downloading github.com/onsi/gocleanup v0.0.0-20140331211545-c1a5478700b5
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4
go: downloading github.com/spf13/cast v1.3.1
go: downloading github.com/ghodss/yaml v1.0.0
go: downloading github.com/OneOfOne/xxhash v1.2.7
go: downloading github.com/rcrowley/go-metrics v0.0.0-20190826022208-cac0b30c2563
go: downloading github.com/gobwas/glob v0.2.3
go: downloading github.com/yashtewari/glob-intersection v0.0.0-20180916065949-5c77d914dd0b
go: downloading github.com/googleapis/gax-go/v2 v2.0.5Result (correct++ - adds missing lines from other deps, adds both 2 zap go.sum lines):
% git diff
diff --git a/go.mod b/go.mod
index ce5e0062..abeba3a5 100644
--- a/go.mod
+++ b/go.mod
@@ -57,7 +57,7 @@ require (
github.com/stretchr/testify v1.6.1
github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80
go.opencensus.io v0.22.4
- go.uber.org/zap v1.15.0
+ go.uber.org/zap v1.16.0
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a
golang.org/x/net v0.0.0-20200904194848-62affa334b73
golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43
diff --git a/go.sum b/go.sum
index 5194c126..a5efb33b 100644
--- a/go.sum
+++ b/go.sum
@@ -14,6 +14,7 @@ cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKV
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0 h1:RmDygqvj27Zf3fCQjQRtLyC7KwFcHkeJitcO0OoGOcA=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
+cloud.google.com/go v0.65.0 h1:Dg9iHVQfrhq82rUNu9ZxUDrJLaxFUe/HlCVaLyRruq8=
cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
@@ -524,6 +525,8 @@ go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9E
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
go.uber.org/zap v1.15.0 h1:ZZCA22JRF2gQE5FoNmhmrf7jeJJ2uhqDUNRYKm8dvmM=
go.uber.org/zap v1.15.0/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc=
+go.uber.org/zap v1.16.0 h1:uFRZXykJGK9lLY4HtgSw44DnIcAM+kRBP7x5m+NpAOM=
+go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ=
golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
@@ -605,6 +608,7 @@ golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20200904194848-62affa334b73 h1:MXfv8rhZWmFeqX3GNZRsd6vOLoaCHjYEX3qkRo3YBUA=
golang.org/x/net v0.0.0-20200904194848-62affa334b73/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -612,6 +616,7 @@ golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4Iltr
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
+golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43 h1:ld7aEMNHoBnnDAX15v1T6z31v8HwR2A9FYOuAhWqkwc=
golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -785,6 +790,7 @@ google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7Fc
google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d h1:92D1fum1bJLKSdr11OJ+54YeCMCGYIygTA7R/YZxH5M=
google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.0/go.mod h1:chYK+tFQF0nDUGJgXMSgLCQk3phJEuONr2DCgLDdAQM=To your point about go mod x commands being a bit non-deterministic, it seems even a tidy in this state locally won't remove the old deps from the go.sum. I am not sure why offhand. It does look like just go get -d should have resulted in this type of diff, though.
Your test repo looks like it matches - https://github.com/renovate-tests/pomerium/pull/1/files. I don't see why the original PR (and the several before it) didn't have the same change set.
Is it somehow possible that several of these running at the same time create issues in the app backend? It looks a bit like we could be sharing a cache from that docker command. I don't know if those go commands are safe for concurrent execution.
travisgroth
commented
4 years ago Is it somehow possible that several of these running at the same time create issues in the app backend?
I just realized we wound up with 1 PR left from the set that got opened at approximately the same time this week. Due to issues outside of renovate's control, we had to close it unmerged. However, that means renovate would have retried just that one change in isolation. The final diff looks like a normal run of go get -d ./..., and it repaired go.sum like I'd expect.
https://github.com/pomerium/pomerium/pull/1382/files
travisgroth
commented
4 years ago Concurrency issues wouldn't explain https://github.com/travisgroth/renovate-test/pull/2/files, unfortunately.
rarkins
commented
4 years ago We don't have any concurrency of requests on the workers too, although there are times when a cache can be left over from past repos.
One thing I'm suspecting is that we don't detect failure every time if the go.sum is updated. e.g. it starts updating go.sum, then exits with an error. So in that case we'd fail to flag the PR as having failed artifacts. This could explain your PR.
The next question is why we have regular failures instead of only occasional. We set GOPROXY=direct because it was failing for people with private modules (IIRC) but maybe by doing everything direct then we're getting rate limited somewhere?
rarkins
commented
4 years ago I looked into my suspicion but did not have it confirmed. Gomod artifact updating here. Exec routine here. Exec appears to catch any error and throw it again, while the gomod logic catches and returns an artifact error. There should be no scenario where the exec throws yet we still update the go.sum. However is there the possibility that go X does not exit with a non-zero, or that somehow our concatenation of commands swallows the non-zero exit code of one command?
rarkins
commented
4 years ago One other thing that's currently unexplained is the Found updated go.mod after go.sum update message in the logs I pasted above. That's meant to indicate that the following happened:
go X commandsgo.mod was modified by the above go X commandsBut in this case go.mod has only one line changed in total, so how was it updated? This could be an unrelated bug, but worth looking into.
rarkins
commented
4 years ago Finally for now, my suspicion is that this problem comes somehow from our go get command and not that go mod tidy isn't tidying.
travisgroth
commented
4 years ago @rarkins first, thanks for following up on this.
Finally for now, my suspicion is that this problem comes somehow from our go get command and not that go mod tidy isn't tidying.
I agree here. I'm at a loss as to how it happens, and so consistently. It did seem to start at the end of July. Were there any changes related to either go support in renovate or the underlying go version you run that coincide with that timeline?
rarkins
commented
4 years ago This change was June 3: https://github.com/renovatebot/renovate/pull/6419 This refactor was August 5: https://github.com/renovatebot/renovate/pull/6912
BTW we are making a change right now to use the default GOPROXY settings as per discussion in #7233. I think this should reduce the frequency at which our go get fails because it will use the more efficient proxied results instead of directly downloading them all.
travisgroth
commented
4 years ago Hmm. We saw the problem start here https://github.com/pomerium/pomerium/pull/1184/files. Before #6912 and well after #6419.
re: GOPROXY. Interesting. Happy to see what happens after.
Edit: when will that change be live?
travisgroth
commented
4 years ago FYI still inconsistent.
https://github.com/pomerium/pomerium/pull/1404/files - missing a line in go.sum. https://github.com/pomerium/pomerium/pull/1406/files - good diff.
Running go mod tidy does fix #1404. Which maybe happened as part of https://github.com/pomerium/pomerium/pull/1405/files.
I'm unsure what to make of it.
rarkins
commented
4 years ago Are the first two the same link? They show up as the same on mobile at least
travisgroth
commented
4 years ago No, two different PRs.
rarkins
commented
4 years ago Does running go get ./.. also fix 1404? Or is it specifically needing a tidy? I still can't work out from the logs what is going "different" between a successful and a non
travisgroth
commented
4 years ago Does running go get ./.. also fix 1404
Yes it looks like it does.
rarkins
commented
4 years ago This problem is really a brain bender. e.g. we have go apparently exiting cleanly yet files are non-reproducibly incomplete. Also it's not like go.sum is not being updated at all.. it's partially updated. If such missing lines should be added by go get ./.. then it seems it's not because go mod tidy failed to add them, but potentially go mod tidy removed it?
Can you explain more on what you mean by the following?
Which maybe happened as part of pomerium/pomerium#1405 (files).
And to clarify, you got 3 PRs - 1404, 1405, 1406 - and were two good and one bad?
FWIW I'm starting to suspect if somehow the cache affects the result. Therefore want to check which was good/bad and if they all ran on the same machine right after each other. And if 1404 was good and 1405/1406 bad then did you see the same pattern previously with the first PR incomplete and later ones complete?
travisgroth
commented
3 years ago Can you explain more on what you mean by the following?
Which maybe happened as part of pomerium/pomerium#1405 (files).
go mod tidy (presumably running at the end) but clearly didn't happen.go mod tidy, but maybe the main go mod command did. And to clarify, you got 3 PRs - 1404, 1405, 1406 - and were two good and one bad?
Yes, and 1405 actually fixed 1404 when it got rebased, I believe.
HonkingGoose
commented
3 years ago Closing this issue as it's a duplicate of issue #6213.
viceice
commented
3 years ago Duplicate of #6213
What Renovate type, platform and version are you using?
Github App
Describe the bug
In recent updates, we're seeing that updates aren't complete in
go.sum, and it doesn't seem likego mod tidyis being run afterwards either.go.sumfile is not being added with the update.go mod tidynormally resolves this state.go mod tidythe final commit by renovate is still missing said line ingo.sumThis PR is an example: https://github.com/pomerium/pomerium/pull/1381/files
There is a missing line which should be present and would have been restored by
go mod tidy:You can see we resolved this (from a number of updates) in this PR by running
go mod tidy: https://github.com/pomerium/pomerium/pull/1384/files. It is the same type of line missing from each previous renovate PR.The strange thing is that running
go get -u [module]does not leave such a state. I've tried it on go1.14.0,1.14.5, and1.15.1. The resulting changes are what you'd expect:I see similar behavior if I manually replace the version in
go.modand rungo get -d ./...Relevant debug logs
Debug Logs from https://github.com/pomerium/pomerium/pull/1381/files
``` DEBUG: processBranch with 1 upgrades(branch="renovate/go.uber.org-zap-1.x") DEBUG: Setting current branch to master(branch="renovate/go.uber.org-zap-1.x") DEBUG: latest commit(branch="renovate/go.uber.org-zap-1.x") { "branchName": "master", "latestCommitDate": "2020-09-04T10:37:00-06:00" } DEBUG: getBranchPr(renovate/go.uber.org-zap-1.x)(branch="renovate/go.uber.org-zap-1.x") DEBUG: findPr(renovate/go.uber.org-zap-1.x, undefined, open)(branch="renovate/go.uber.org-zap-1.x") DEBUG: branchExists=false(branch="renovate/go.uber.org-zap-1.x") DEBUG: Branch has 1 upgrade(s)(branch="renovate/go.uber.org-zap-1.x") DEBUG: recreateClosed is false(branch="renovate/go.uber.org-zap-1.x") DEBUG: findPr(renovate/go.uber.org-zap-1.x, chore(deps): update module go.uber.org/zap to v1.16.0, !open)(branch="renovate/go.uber.org-zap-1.x") DEBUG: prAlreadyExisted=false(branch="renovate/go.uber.org-zap-1.x") DEBUG: Checking schedule(before 3am on Monday, null)(branch="renovate/go.uber.org-zap-1.x") DEBUG: Checking 1 schedule(s)(branch="renovate/go.uber.org-zap-1.x") DEBUG: Checking schedule "before 3am on Monday"(branch="renovate/go.uber.org-zap-1.x") { "parsedSchedule": { "schedules": [ { "t_b": [ 10800 ], "d": [ 2 ] } ], "exceptions": [], "error": -1 } } DEBUG: Matches schedule before 3am on Monday(branch="renovate/go.uber.org-zap-1.x") DEBUG: Branch needs creating(branch="renovate/go.uber.org-zap-1.x") DEBUG: Using reuseExistingBranch: false(branch="renovate/go.uber.org-zap-1.x") DEBUG: manager.getUpdatedPackageFiles()(branch="renovate/go.uber.org-zap-1.x") { "reuseExistingBranch": false, "branchName": "renovate/go.uber.org-zap-1.x" } DEBUG: gomod.updateDependency: v1.16.0(branch="renovate/go.uber.org-zap-1.x") DEBUG: Updating packageFile content(packageFile="go.mod", branch="renovate/go.uber.org-zap-1.x") { "depName": "go.uber.org/zap" } DEBUG: gomod.updateArtifacts(go.mod)(branch="renovate/go.uber.org-zap-1.x") DEBUG: Using GOPATH: /tmp/renovate-cache/others/go(branch="renovate/go.uber.org-zap-1.x") DEBUG: go get command included(branch="renovate/go.uber.org-zap-1.x") { "cmd": "go", "args": "get -d ./..." } DEBUG: go mod tidy command included(branch="renovate/go.uber.org-zap-1.x") { "cmd": "go", "args": "mod tidy" } DEBUG: additional go mod tidy command included(branch="renovate/go.uber.org-zap-1.x") { "cmd": "go", "args": "mod tidy" } DEBUG: Using docker to execute(branch="renovate/go.uber.org-zap-1.x") DEBUG: Found npm version constraint - checking for a compatible renovate/go image to use(branch="renovate/go.uber.org-zap-1.x") { "constraint": "1.14" } DEBUG: Missing versioning(branch="renovate/go.uber.org-zap-1.x") DEBUG: Found compatible npm version(branch="renovate/go.uber.org-zap-1.x") { "constraint": "1.14", "version": "1.14.5" } DEBUG: Resolved tag constraint(branch="renovate/go.uber.org-zap-1.x") { "image": "renovate/go", "tagConstraint": "1.14", "tagVersioning": "npm", "tag": "1.14.5" } DEBUG: Docker image is already prefetched: renovate/go:1.14.5(branch="renovate/go.uber.org-zap-1.x") DEBUG: Executing command(branch="renovate/go.uber.org-zap-1.x") { "command": "docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/pomerium/pomerium\":\"/mnt/renovate/gh/pomerium/pomerium\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/pomerium/pomerium\" renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://github.com/\\\" && go get -d ./... && go mod tidy && go mod tidy\"" } DEBUG: exec completed(branch="renovate/go.uber.org-zap-1.x") { "cmd": "docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/pomerium/pomerium\":\"/mnt/renovate/gh/pomerium/pomerium\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/pomerium/pomerium\" renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://github.com/\\\" && go get -d ./... && go mod tidy && go mod tidy\"", "durationMs": 9164, "stdout": "", "stderr": "" } DEBUG: Returning updated go.sum(branch="renovate/go.uber.org-zap-1.x") DEBUG: Updated 1 package files(branch="renovate/go.uber.org-zap-1.x") DEBUG: Getting updated lock files(branch="renovate/go.uber.org-zap-1.x") DEBUG: lock file dirs(branch="renovate/go.uber.org-zap-1.x") { "dirs": { "yarnLockDirs": [], "npmLockDirs": [], "pnpmShrinkwrapDirs": [], "lernaDirs": [] } } DEBUG: Writing package.json files(branch="renovate/go.uber.org-zap-1.x") { "packageFiles": [ "package.json" ] } DEBUG: Writing any updated package files(branch="renovate/go.uber.org-zap-1.x") DEBUG: Updated 1 lock files(branch="renovate/go.uber.org-zap-1.x") { "updatedArtifacts": [ "go.sum" ] } DEBUG: 2 file(s) to commit(branch="renovate/go.uber.org-zap-1.x") DEBUG: Committing files to branch renovate/go.uber.org-zap-1.x(branch="renovate/go.uber.org-zap-1.x") INFO: Branch created(branch="renovate/go.uber.org-zap-1.x") { "commitSha": "b26c364" } DEBUG: Ensuring PR(branch="renovate/go.uber.org-zap-1.x") DEBUG: There are 0 errors and 0 warnings(branch="renovate/go.uber.org-zap-1.x") DEBUG: getBranchPr(renovate/go.uber.org-zap-1.x)(branch="renovate/go.uber.org-zap-1.x") DEBUG: findPr(renovate/go.uber.org-zap-1.x, undefined, open)(branch="renovate/go.uber.org-zap-1.x") DEBUG: getBranchStatus(renovate/go.uber.org-zap-1.x)(branch="renovate/go.uber.org-zap-1.x") DEBUG: branch status check result(branch="renovate/go.uber.org-zap-1.x") { "state": "pending", "statuses": [] } DEBUG: No check runs found(branch="renovate/go.uber.org-zap-1.x") { "result": { "total_count": 0, "check_runs": [] } } DEBUG: Creating PR(branch="renovate/go.uber.org-zap-1.x") { "prTitle": "chore(deps): update module go.uber.org/zap to v1.16.0" } DEBUG: Creating PR(branch="renovate/go.uber.org-zap-1.x") { "title": "chore(deps): update module go.uber.org/zap to v1.16.0", "head": "pomerium:renovate/go.uber.org-zap-1.x", "base": "master", "draft": false } DEBUG: PR created(branch="renovate/go.uber.org-zap-1.x") { "pr": 1381, "draft": false } DEBUG: Adding labels 'dependency' to #1381(branch="renovate/go.uber.org-zap-1.x") INFO: PR created(branch="renovate/go.uber.org-zap-1.x") { "pr": 1381, "prTitle": "chore(deps): update module go.uber.org/zap to v1.16.0" } DEBUG: getBranchStatus(renovate/go.uber.org-zap-1.x)(branch="renovate/go.uber.org-zap-1.x") DEBUG: branch status check result(branch="renovate/go.uber.org-zap-1.x") { "state": "pending", "statuses": [] } DEBUG: No check runs found(branch="renovate/go.uber.org-zap-1.x") { "result": { "total_count": 0, "check_runs": [] } } DEBUG: getBranchStatus() result(branch="renovate/go.uber.org-zap-1.x") { "branchStatus": "yellow", "branchName": "renovate/go.uber.org-zap-1.x" } DEBUG: Skipping assignees and reviewers as automerge=true(branch="renovate/go.uber.org-zap-1.x") DEBUG: Created Pull Request #1381(branch="renovate/go.uber.org-zap-1.x") DEBUG: Getting comments for #1381(branch="renovate/go.uber.org-zap-1.x") DEBUG: Found 0 comments(branch="renovate/go.uber.org-zap-1.x") DEBUG: Checking #1381 for automerge(branch="renovate/go.uber.org-zap-1.x") { "automerge": true, "automergeType": "pr", "automergeComment": "automergeComment" } DEBUG: PR is configured for automerge(branch="renovate/go.uber.org-zap-1.x") DEBUG: PR is not ready for merge(branch="renovate/go.uber.org-zap-1.x") ```To Reproduce
gomodTidyset: https://github.com/travisgroth/renovate-testgomodTidynot set: https://github.com/travisgroth/renovate-test-tidyBoth of these repos are one patch behind a viper dependency.
Additional context Our updates are weekly.