Renovate bot doesn't add package sum entry to go.sum

[at-wat]

Generating package sum requires full source and it is resource consuming job; Renovate bot looks not doing it for all package. ref: https://github.com/renovatebot/renovate/issues/3017

e.g. https://github.com/pion/dtls/commit/9180e9292dad450797116b1f3a8d8c4fa2e94978

- golang.org/x/crypto v0.0.0-20191205180655-e7c4368fe9dd h1:GGJVjV8waZKRHrgwvtH66z9ZGVurTD1MT0n1Bb+q4aM=
- golang.org/x/crypto v0.0.0-20191205180655-e7c4368fe9dd/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+ golang.org/x/crypto v0.0.0-20191227163750-53104e6ec876/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=

which should be

- golang.org/x/crypto v0.0.0-20191205180655-e7c4368fe9dd h1:GGJVjV8waZKRHrgwvtH66z9ZGVurTD1MT0n1Bb+q4aM=
- golang.org/x/crypto v0.0.0-20191205180655-e7c4368fe9dd/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+ golang.org/x/crypto v0.0.0-20191227163750-53104e6ec876 h1:sKJQZMuxjOAR/Uo2LBfU90onWEf1dF4C+0hPJCc9Mpc=
+ golang.org/x/crypto v0.0.0-20191227163750-53104e6ec876/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=

It causes an error on Go 1.13. https://travis-ci.org/pion/dtls/builds/636289709

verifying golang.org/x/crypto@v0.0.0-20191227163750-53104e6ec876: golang.org/x/crypto@v0.0.0-20191227163750-53104e6ec876: open /go/pkg/sumdb/sum.golang.org/latest: no such file or directory

Adding a CI job to do go mod download, go mod tidy, commit and push would fix it.

[at-wat]

creating https://github.com/at-wat/go-sum-fix-action

[at-wat]

CI is not triggered by push from actions. go-sum-fix-action doesn't help.

[at-wat]

Now, CI webhooks seems be changed to be triggered by push by Actions Bot. GitHub Actions are not triggered to prevent endless recursive call.

[vidavidorra]

Hi @at-wat Just saw this issue when I was looking at the workings of this repo. I've been using Renovate a lot lately and was working on some project with it. I just wanted to mention here that Renovate has the postUpdateOptions, which you can set to gomodTidy to achieve running go mod tidy after its updates, but before the commit I believe. Additionally it also has a postUpgradeTasks, which can be used to specify commands to be run, so that should be able to run go mod download.

Note that I haven't tested this at all, but I believe it should be able to apply the fixes you described in this ticket, without the need of an extra GA workflow that has to be run after the update, and also the issue you describe about the prevention of endless recursive call. (I think that is the fact that Github Actions workflows are not allowed to trigger other workflows to prevent endless recursive runs.)

[at-wat]

Yea, we already set gomodTidy, but I didn't know postUpgradeTasks. (It is known issue of Renovate that package sum is sometimes lost even if set gomodTidy. https://github.com/renovatebot/renovate/issues/3017) Could you make a PR to this repository?

(I think that is the fact that Github Actions workflows are not allowed to trigger other workflows to prevent endless recursive runs.)

Right, Actions are not triggered by Actions, but actually other CIs are triggered by Actions at now, as shown in https://github.com/pion/.goassets/issues/14#issuecomment-596527681.

[vidavidorra]

Yea, we already set gomodTidy, but I didn't know postUpgradeTasks. Could you make a PR to this repository?

Sure! If I understood correctly, the workflow (https://github.com/pion/.goassets/blob/master/ci/.github/workflows/renovate-go-mod-fix.yaml) can be removed that right?

I'm not familiar with the go commands (except go mod tidy). I don't know the order that postUpgradeTasks and gomodTidy run in. Does that matter for this task? I.e. must go mod download be run before go mod tidy?

If it matters, I think both need to be run using postUpgradeTasks or I need to check the docs further for the order (and ask the guys at Renovate what the order is if I can't find it in the docs).

[at-wat]

Yes, if it works, the workflow can be removed.

I.e. must go mod download be run before go mod tidy?

Yes, the command must be in the order of go mod download then go mod tidy.

[vidavidorra]

Cool! I'll work on this either tonight or tomorrow and check with the guys at Renovate for the order if needed. I'm reopening this to track it ;)

[vidavidorra]

I've asked the question to Renovate, but apparently postUpgradeTasks isn't available from the app (probably for security reasons). Another thing Rhys noted is that gomodTidy actually runs runs go get -d ./... before go mod tidy, see discussion here. I've looked at the go docs and go get -d has the following documentation.

The -d flag instructs get to download the source code needed to build the named packages, including downloading necessary dependencies, but not to build and install them.

I've also looked at the docs for go mod download and it has the following. I'm not familiar with these commands, but based on the docs they look similar. I'd have to dig deeper in the logs to see whether only gomodTidy would be enough, but at this point I'm not sure whether I can easily fix this.

Download downloads the named modules, which can be module patterns selecting dependencies of the main module or module queries of the form path@version. With no arguments, download applies to all dependencies of the main module.

[at-wat]

I'll release the Actions workflow for now. Let's continuously search better way.

[at-wat]

The action works as expected. https://github.com/pion/dtls/pull/241

[at-wat]

go-mod-fix action is now working as expected. Please reopen if anyone has alternative way.

Originally, this issue is caused by the limitation of the renovate public server resource. (It doesn't clone whole package, but only go.mod.) If the renovate server supports to fully clone all dependencies or we host our own renovate server, the action can be removed. (As a reference, one of the alternative service Dependabot seems not yet supporting go mod tidy.)