search

renovatebot / renovate

Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io
https://mend.io/renovate
GNU Affero General Public License v3.0
17.29k stars 2.26k forks source link

Major version update of Go module is not detected when GOPROXY is set #16663

Closed at-wat closed 1 year ago

at-wat commented 2 years ago

How are you running Renovate?

Mend Renovate hosted app on github.com

Please select which platform you are using if self-hosting.

github.com

Was this something which used to work for you, and then stopped?

It used to work, and then stopped

Describe the bug

Minimal reproduction: https://github.com/at-wat/renovate-repro-major-not-updated

Reproduction repository has dependency to github.com/google/go-github/v44 v44.0.0. New versions v44.1.0 and v45.2.0 are available.

Expected behavior: Major version update PR (Update module github.com/google/go-github/v44 to v45) and minor version update PR (Update module github.com/google/go-github/v44 to v44.1.0) is created. Actual behavior: Major version update PR of Go module is not created. Only minor version update PR (Update module github.com/google/go-github/v44 to v44.1.0) is created.

Hosted Renovate doesn't detect major version update of Go module. Self-hosted Renovate detects. (Logs below)

On my private repository, the last major version update PR for github.com/google/go-github was v39 on Oct 4, 2021. v40.0.0 was released on Nov 08, 2021, so something goes wrong in this period.

I tested several version of self-hosted Renovate from 28.3.0 to 32.119.2 and all of them seems detecting major update of github.com/google/go-github. I also added an another Go module (github.com/go-chi/chi/v4 -> github.com/go-chi/chi/v5) to the reproduction repository and same problem occurred. (removed now to make it minimal)

Relevant debug logs

Logs on hosted Renovate (32.117.4) ``` DEBUG: packageFiles with updates { "config": { "gomod": [ { "packageFile": "go.mod", "constraints": { "go": "^1.18" }, "deps": [ { "managerData": { "lineNumber": 4 }, "depName": "github.com/google/go-github/v44", "depType": "require", "currentValue": "v44.0.0", "datasource": "go", "depIndex": 0, "updates": [ { "bucket": "non-major", "newVersion": "v44.1.0", "newValue": "v44.1.0", "releaseTimestamp": "2022-05-13T17:01:22.000Z", "newMajor": 44, "newMinor": 1, "updateType": "minor", "branchName": "renovate/github.com-google-go-github-v44-44.x" } ], "warnings": [], "versioning": "semver", "sourceUrl": "https://github.com/google/go-github", "currentVersion": "v44.0.0", "isSingleVersion": true, "fixedVersion": "v44.0.0" } ] } ] } } ```
Logs on self-hosted Renovate (32.117.4) ``` DEBUG: packageFiles with updates (repository=at-wat/renovate-repro-major-not-updated) "config": { "gomod": [ { "packageFile": "go.mod", "constraints": {"go": "^1.18"}, "deps": [ { "managerData": {"lineNumber": 4}, "depName": "github.com/google/go-github/v44", "depType": "require", "currentValue": "v44.0.0", "datasource": "go", "depIndex": 0, "updates": [ { "bucket": "non-major", "newVersion": "v44.1.0", "newValue": "v44.1.0", "releaseTimestamp": "2022-05-13T17:01:22.000Z", "newMajor": 44, "newMinor": 1, "updateType": "minor", "branchName": "renovate/github.com-google-go-github-v44-44.x" }, { "bucket": "major", "newVersion": "v45.2.0", "newValue": "v45.2.0", "releaseTimestamp": "2022-06-20T10:54:32.000Z", "newMajor": 45, "newMinor": 2, "updateType": "major", "branchName": "renovate/github.com-google-go-github-v44-45.x" } ], "warnings": [], "versioning": "semver", "sourceUrl": "https://github.com/google/go-github", "currentVersion": "v44.0.0", "isSingleVersion": true, "fixedVersion": "v44.0.0" } ] } ] } ```

Have you created a minimal reproduction repository?

I have linked to a minimal reproduction repository in the bug description

viceice commented 2 years ago

Please update the reproduction readme with current and expected behavior

at-wat commented 2 years ago

Added to the description:

Expected behavior: Major version update PR of Go module is created Actual behavior: Major version update PR of Go module is not created

PhilipAbed commented 2 years ago

so we are getting "newMajor": 44, but it should be 45 right?

at-wat commented 2 years ago

On the hosted Renovete, only

[
  {"bucket": "non-major", "newVersion": "v44.1.0", "newMajor": 44}
]

is detected.

But it should be:

[
  {"bucket": "non-major", "newVersion": "v44.1.0", "newMajor": 44},
  {"bucket": "major", "newVersion": "v45.2.0", "newMajor": 45}
]
rarkins commented 2 years ago

The hosted renovate uses the default GOPROXY settings while maybe in self hosted that is not the case and Renovate looks up the GitHub repo tags directly

at-wat commented 2 years ago

I forked it to an organization and doesn't get v45:

```json "updates": [ { "bucket": "non-major", "newVersion": "v44.1.0", "newValue": "v44.1.0", "releaseTimestamp": "2022-05-13T17:01:22.000Z", "newMajor": 44, "newMinor": 1, "updateType": "minor", "branchName": "renovate/github.com-google-go-github-v44-44.x" } ], ```
PhilipAbed commented 2 years ago

i can confirm it works on self hosted https://github.com/StinkyLord/renovate-repro-major-not-updated/pulls

at-wat commented 2 years ago

I just set GOPROXY=https://proxy.golang.org to the self-hosted one then v45 disappeared as same as the hosted:

``` renovate_1 | "updates": [ renovate_1 | { renovate_1 | "bucket": "non-major", renovate_1 | "newVersion": "v44.1.0", renovate_1 | "newValue": "v44.1.0", renovate_1 | "releaseTimestamp": "2022-05-13T17:01:22.000Z", renovate_1 | "newMajor": 44, renovate_1 | "newMinor": 1, renovate_1 | "updateType": "minor", renovate_1 | "branchName": "renovate/github.com-google-go-github-v44-44.x" renovate_1 | } renovate_1 | ], ```
at-wat commented 2 years ago

Updated issue description:

> Minimal reproduction: https://github.com/at-wat/renovate-repro-major-not-updated > > Reproduction repository has dependency to `github.com/google/go-github/v44 v44.0.0`. > New versions `v44.1.0` and `v45.2.0` are available. > > **Expected behavior**: > Major version update PR (`Update module github.com/google/go-github/v44 to v45`) and minor version update PR (`Update module github.com/google/go-github/v44 to v44.1.0`) is created. > **Actual behavior**: > Major version update PR of Go module is not created. > Only minor version update PR (`Update module github.com/google/go-github/v44 to v44.1.0`) is created.

and updated docker-compose.yml in the reproduction repo to reproduce it locally.

at-wat commented 2 years ago

I'm playing around locally and go datasource with GOPROXY env variable seems querying:

Without GOPROXY, GithubTagsDatasource are used and it lists all git tags including v45.*.*.

In the GOPROXY documentation at https://go.dev/ref/mod#goproxy-protocol, there are no public API to query list of the major versions. On actual GOPROXY (proxy.golang.org),

rarkins commented 2 years ago

Seems no elegant way to learn of all major versions through the proxy?

at-wat commented 2 years ago

As far as I looked into, there isn't.

Even in the pkg.go.dev's source code, they do use GOPROXY elsewhere, but the page listing major versions (*) uses RDB as a datasource.

(*) image

at-wat commented 2 years ago

I'm trying to fix this unusing GOPROXY without relapsing #8852.

By using GitTags/GitRefs as a base datasource, we can filter the list of tags by git ref pattern. https://git-scm.com/docs/git-ls-remote.html#Documentation/git-ls-remote.txt-ltrefsgt82308203

Go modules causing #8852 which have too many tags are usually having many nested modules. As each nested modules should have submodule/vX.Y.Z tags, number of the total tags explodes. Adding filter pattern like refs/tags/submodule/v* refs/tags/v* dramatically decreases number of the response items in such case and major version updates are retrieved as well.

rarkins commented 2 years ago

What about modules which aren't hosted on GitHub?

at-wat commented 2 years ago

Root git repository of the Go module is discovered by https://module-url?go-get=1. It should work for all go modules which can be discovered by real Go command.

rarkins commented 2 years ago

If we use git-refs to get the full list of tags, is there any advantage to still using GOPROXY at all? e.g. does it make anything simpler for us or may as well we just implement the logic ourselves?

at-wat commented 2 years ago

It should work for all go modules which can be discovered by real Go command.

I noticed that the current code of my PR doesn't support go modules managed in non-git repositories. (Go mod supports Bazaar, Fossil, Git, Mercurial, and Subversion: https://go.dev/ref/mod#vcs-find) I think current go datasource without GOPROXY also doesn't support them.

One advantage of GOPROXY is that we don't need to handle underlying VCS type.

If we use git-refs to get the full list of tags, is there any advantage to still using GOPROXY at all?

If git ls-remote supported listing only one tag for each major version, combining with GOPROXY might be good at performance, but I think it's not possible due to the spec of ls-remote command. Another option to use GOPROXY would be making a DB storing all major versions of every go modules just like pkg.go.dev does.

rarkins commented 2 years ago

Maybe we could have an algorithm that tries listing releases of a major version bump (Eg v45 in your example) from GOPROXY and if it fails then we remember it for a day so as not to retry. Obviously we can keep bumping until we get a negative response. Alternatively we propose to the go team adding this listing capability to their proxy natively

at-wat commented 2 years ago

Obviously we can keep bumping until we get a negative response.

I also considered it once, but it works only if the major versions has no gap.

About the go modules design, versions with different major versions are basically considered as different modules to make it possible to import multiple major versions of the same path at same time. So, I guess the chance to add such feature to GOPROXY is little low. (We can ask Go team anyway)

renovate-release commented 1 year ago

:tada: This issue has been resolved in version 34.66.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: