search

renovatebot / renovate

Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io
https://mend.io/renovate
GNU Affero General Public License v3.0
17.29k stars 2.26k forks source link

Package lookup fails with Golang submodules In Gitlab.com #28540

Open rarkins opened 5 months ago

rarkins commented 5 months ago

Discussed in https://github.com/renovatebot/renovate/discussions/28530

Originally posted by **rrey** April 19, 2024 ### What would you like help with? I think I found a bug ### How are you running Renovate? Self-hosted ### If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate. GitLab and renovate 37.308.0 ### Please tell us more about your question or problem I have private repositories on gitlab.com where I reproduce the error, let me know who I shall add to the repository to allow you to reproduce. I have a project with the following `go.mod`: ``` module [gitlab.com/rrey2/some-project](https://gitlab.com/rrey2/some-project/) go 1.18 require ( [gitlab.com/rrey2/some-deps](https://gitlab.com/rrey2/some-deps/) v1.0.0 [gitlab.com/rrey2/some-deps/submodule](https://gitlab.com/rrey2/some-deps/submodule/) v1.0.0 ) ``` The first module `gitlab.com/rrey2/some-deps` is not an issue and renovate can process it. The second one is a submodule located in the same repository as the first one in a subdirectory. For the submodule the lookup fails. See attached logs. ### Logs (if relevant)
Logs ``` DEBUG: Fetching gitlab.com/rrey2/some-deps via GONOPROXY match (repository=rrey2/some-project) DEBUG: Fetching gitlab.com/rrey2/some-deps/submodule via GONOPROXY match (repository=rrey2/some-project) DEBUG: Go lookup source url https://gitlab.com/rrey2/some-deps for module gitlab.com/rrey2/some-deps (repository=rrey2/some-project) DEBUG: Extracted information: gitlabUrl: https://gitlab.com, gitlabUrlName: rrey2/some-deps, gitlabModuleName: rrey2/some-deps (repository=rrey2/some-project) DEBUG: Go lookup source url https://gitlab.com/rrey2/some-deps for module gitlab.com/rrey2/some-deps/submodule (repository=rrey2/some-project) DEBUG: Extracted information: gitlabUrl: https://gitlab.com, gitlabUrlName: rrey2/some-deps, gitlabModuleName: rrey2/some-deps/submodule (repository=rrey2/some-project) DEBUG: GET https://gitlab.com/api/v4/projects/rrey2%2Fsome-deps%2Fsubmodule/repository/tags?per_page=100 = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=164) (repository=rrey2/some-project) DEBUG: GitLab API 404 (repository=rrey2/some-project) "url": "https://gitlab.com/api/v4/projects/rrey2%2Fsome-deps%2Fsubmodule/repository/tags?per_page=100" DEBUG: Datasource 404 (repository=rrey2/some-project) "datasource": "go", "packageName": "gitlab.com/rrey2/some-deps/submodule", "url": "https://gitlab.com/api/v4/projects/rrey2%2Fsome-deps%2Fsubmodule/repository/tags?per_page=100" DEBUG: Failed to look up go package gitlab.com/rrey2/some-deps/submodule (repository=rrey2/some-project, packageFile=go.mod, dependency=gitlab.com/rrey2/some-deps/submodule) ```
The problem seems to be that the URL used to fetch the submodule is not the repository url, while the response from gitlab to the `go-get=1` properly points to the repository: ``` $ curl 'https://gitlab.com/rrey2/some-deps/submodule?go-get=1' go get https://gitlab.com/rrey2/some-deps ``` I tried to debug by pulling the repo and testing on `main` by adding debug logs. I added a log in `lib/modules/datasource/go/base.ts` to show what is the parsed content of the meta blocks. These logs are visible in my extract as the lines starting with `Extracted information`. We can see that the `gitlabUrlName` variable properly points to the repository, but the GET is not performed on this path.
rarkins commented 5 months ago

Unit test: https://github.com/renovatebot/renovate/pull/28533

addudko commented 3 months ago

Hi Rhys (@rarkins), I'm not sure if this was fixed, but could you please also include this scenario in the issue:

module gitlab.com/rrey2/some-project

go 1.22.2

require (
    gitlab.com/rrey2/some-subgroup/repo-with-gomod v1.0.0
    gitlab.com/rrey2/some-subgroup/repo-with-gomod/submodule-folder v0.1.0
)

In renovate logs this case looks like:

DEBUG: Fetching gitlab.com/rrey2/some-subgroup/repo-with-gomod/submodule-folder via GONOPROXY match (repository=rrey2/some-project)
DEBUG: Fetching gitlab.com/rrey2/some-subgroup/repo-with-gomod via GONOPROXY match (repository=rrey2/some-project)
DEBUG: Go lookup source url https://gitlab.com/rrey2/some-subgroup for module gitlab.com/rrey2/some-subgroup/repo-with-gomod/submodule-folder (repository=rrey2/some-project)
DEBUG: Go lookup source url https://gitlab.com/rrey2/some-subgroup for module gitlab.com/rrey2/some-subgroup/repo-with-gomod (repository=rrey2/some-project)
DEBUG: GET https://gitlab.com/api/v4/projects/rrey2%2Fsome-subgroup/repository/tags?per_page=100 = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=378) (repository=rrey2/some-project)
DEBUG: GitLab API 404 (repository=rrey2/some-project)
       "url": "https://gitlab.com/api/v4/projects/rrey2%2Fsome-subgroup/repository/tags?per_page=100"
DEBUG: GitLab API 404 (repository=rrey2/some-project)
       "url": "https://gitlab.com/api/v4/projects/rrey2%2Fsome-subgroup/repository/tags?per_page=100"
DEBUG: Datasource 404 (repository=rrey2/some-project)
       "datasource": "go",
       "packageName": "gitlab.com/rrey2/some-subgroup/repo-with-gomod/submodule-folder",
       "url": "https://gitlab.com/api/v4/projects/rrey2%2Fsome-subgroup/repository/tags?per_page=100"
DEBUG: Datasource 404 (repository=rrey2/some-project)
       "datasource": "go",
       "packageName": "gitlab.com/rrey2/some-subgroup/repo-with-gomod",
       "url": "https://gitlab.com/api/v4/projects/rrey2%2Fsome-subgroup/repository/tags?per_page=100"
DEBUG: Failed to look up go package gitlab.com/rrey2/some-subgroup/repo-with-gomod/submodule-folder (repository=rrey2/some-project, packageFile=go.mod, dependency=gitlab.com/rrey2/some-subgroup/repo-with-gomod/submodule-folder)
DEBUG: Failed to look up go package gitlab.com/rrey2/some-subgroup/repo-with-gomod (repository=rrey2/some-project, packageFile=go.mod, dependency=gitlab.com/rrey2/some-subgroup/repo-with-gomod)

If try to directly call the same url - you also get 404, but if you add after sub-group the repository name - you'll get the correct page:

https://gitlab.com/api/v4/projects/rrey2%2Fsome-subgroup%2Frepo-with-gomod/repository/tags?per_page=100

Gitlab is self-hosted, our version includes this fix https://gitlab.com/gitlab-org/gitlab/-/issues/36354. Renovate version 37.381.6.

Thank you in advance!

github-actions[bot] commented 3 months ago

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

Please follow these steps:

  1. Read our guide on creating a minimal reproduction.
  2. Go to our minimal reproduction template repository.
  3. Select the Use this template button to create a new repository based on the template.
  4. Work on the minimal reproduction in your own repository.
  5. Fill out the information in your repository's README.md.
  6. Add the link to your reproduction to the first post of your Discussion. If you are not the original author, you can post a new comment with the link.

Good luck,

The Renovate team

rarkins commented 3 months ago

@addudko could you put this into a reproduction on github.com?

addudko commented 3 months ago

@rarkins, I've prepared a minimum reproduction repo: https://github.com/addudko/renovate-issue-28540 In fact, bot correctly works with module that is in subgroup, however the issue is still exists for submodule. So, basically I've prepared minimum reproducible repo exactly for your case :)

addudko commented 3 months ago

Now my issue with subgroup project looks like a matter of access. Could you please tell me what can be missing for our Renovate configuration, if I provided to bot gitlab group access token with the following scopes: api, read_api, read_repository, write_repository?

rarkins commented 3 months ago

Do you mean it's a private project/repo?

addudko commented 3 months ago

Yes, at this example https://github.com/addudko/renovate-issue-28540 it uses public repos, but on my project we use private repo as a module.

rarkins commented 3 months ago

When I run pnpm start --dry-run=lookup addudko/renovate-issue-28540 I don't get the same log messages as in the reproduction.

Failing to look these up in proxy.golang.org isn't necessarily a problem:

DEBUG: GET https://proxy.golang.org/gitlab.com/test-nested/services/services-subgroup/module-test/v2/@v/list = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=3621) (repository=addudko/renovate-issue-28540)
DEBUG: Go lookup source url https://gitlab.com/test-nested/services/services-subgroup/module-test for module gitlab.com/test-nested/services/services-subgroup/module-test (repository=addudko/renovate-issue-28540)
DEBUG: Found no satisfying versions with 'loose' versioning (repository=addudko/renovate-issue-28540)
DEBUG: Go lookup source url https://gitlab.com/test-nested/services/services-subgroup/module-test for module gitlab.com/test-nested/services/services-subgroup/module-test (repository=addudko/renovate-issue-28540)
DEBUG: GET https://proxy.golang.org/gitlab.com/test-nested/services/services-subgroup/module-test/api/v2/@v/list = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=2790) (repository=addudko/renovate-issue-28540)
DEBUG: Go lookup source url https://gitlab.com/test-nested/services/services-subgroup/module-test for module gitlab.com/test-nested/services/services-subgroup/module-test/api (repository=addudko/renovate-issue-28540)

Then both appear to have results:

    {
                 "datasource": "go",
                 "depType": "require",
                 "depName": "gitlab.com/test-nested/services/services-subgroup/module-test",
                 "currentValue": "v0.0.0-20240621083552-75992df8e8a1",
                 "currentDigest": "75992df8e8a1",
                 "digestOneAndOnly": true,
                 "versioning": "loose",
                 "managerData": {"multiLine": true, "lineNumber": 5},
                 "updates": [
                   {
                     "bucket": "non-major",
                     "newVersion": "v0.0.1",
                     "newValue": "v0.0.1",
                     "releaseTimestamp": "2024-06-21T08:58:57.000Z",
                     "newMajor": 0,
                     "newMinor": 0,
                     "newPatch": 1,
                     "updateType": "patch",
                     "newDigest": "2d26b8c1cf509f491d5d83827afe9d548a9951f9",
                     "branchName": "renovate/gitlab.com-test-nested-services-services-subgroup-module-test-0.x"
                   }
                 ],
                 "packageName": "gitlab.com/test-nested/services/services-subgroup/module-test",
                 "warnings": [],
                 "sourceUrl": "https://gitlab.com/test-nested/services/services-subgroup/module-test",
                 "currentVersion": "v0.0.0-20240621083552-75992df8e8a1",
                 "isSingleVersion": true,
                 "fixedVersion": "v0.0.0-20240621083552-75992df8e8a1"
               },
               {
                 "datasource": "go",
                 "depType": "require",
                 "depName": "gitlab.com/test-nested/services/services-subgroup/module-test/api",
                 "currentValue": "v0.0.0-20240621083552-75992df8e8a1",
                 "currentDigest": "75992df8e8a1",
                 "digestOneAndOnly": true,
                 "versioning": "loose",
                 "managerData": {"multiLine": true, "lineNumber": 6},
                 "updates": [
                   {
                     "bucket": "non-major",
                     "newVersion": "v0.0.0-20240621085857-2d26b8c1cf50",
                     "newValue": "v0.0.0-20240621085857-2d26b8c1cf50",
                     "newDigest": "2d26b8c1cf50",
                     "releaseTimestamp": "2024-06-21T08:58:57.000Z",
                     "newMajor": 0,
                     "newMinor": 0,
                     "newPatch": 0,
                     "updateType": "digest",
                     "branchName": "renovate/gitlab.com-test-nested-services-services-subgroup-module-test-api-digest"
                   }
                 ],
                 "packageName": "gitlab.com/test-nested/services/services-subgroup/module-test/api",
                 "warnings": [],
                 "sourceUrl": "https://gitlab.com/test-nested/services/services-subgroup/module-test/api",
                 "currentVersion": "v0.0.0-20240621083552-75992df8e8a1",
                 "isSingleVersion": true,
                 "fixedVersion": "v0.0.0-20240621083552-75992df8e8a1"
               }
             ],
             "packageFile": "go.mod"
           }

http stats:

DEBUG: HTTP statistics (repository=addudko/renovate-issue-28540)
       "urls": {
         "https://api.github.com/graphql": {"POST": {"200": 2}},
         "https://api.github.com/repos/addudko/renovate-issue-28540/pulls": {
           "GET": {"200": 1}
         },
         "https://gitlab.com/api/v4/projects/test-nested%2Fservices%2Fservices-subgroup%2Fmodule-test/repository/commits/v0.0.1": {
           "GET": {"200": 1}
         },
         "https://gitlab.com/test-nested/services/services-subgroup/module-test/api": {
           "GET": {"200": 1}
         },
         "https://gitlab.com/test-nested/services/services-subgroup/module-test": {
           "GET": {"200": 1}
         },
         "https://proxy.golang.org/gitlab.com/test-nested/services/services-subgroup/module-test/@latest": {
           "GET": {"200": 1}
         },
         "https://proxy.golang.org/gitlab.com/test-nested/services/services-subgroup/module-test/@v/list": {
           "GET": {"200": 1}
         },
         "https://proxy.golang.org/gitlab.com/test-nested/services/services-subgroup/module-test/@v/v0.0.1.info": {
           "GET": {"200": 1}
         },
         "https://proxy.golang.org/gitlab.com/test-nested/services/services-subgroup/module-test/api/@latest": {
           "GET": {"200": 1}
         },
         "https://proxy.golang.org/gitlab.com/test-nested/services/services-subgroup/module-test/api/@v/list": {
           "GET": {"200": 1}
         },
         "https://proxy.golang.org/gitlab.com/test-nested/services/services-subgroup/module-test/api/v2/@v/list": {
           "GET": {"404": 1}
         },
         "https://proxy.golang.org/gitlab.com/test-nested/services/services-subgroup/module-test/v2/@v/list": {
           "GET": {"404": 1}
         },
         "https://raw.githubusercontent.com/golang/website/HEAD/internal/history/release.go": {
           "GET": {"200": 1}
         }
       },

So I'm not sure what exactly is the problem

rarkins commented 3 months ago

@addudko please install the Renovate github app and add encrypted hostRules. Then you can access the logs at developer.mend.io to confirm the reproduction

addudko commented 3 months ago

@rarkins I'm sorry, I can't do that due to government restrictions.

bassforce86 commented 2 months ago

Apologies for continuing an open issue - However I believe my issue is related and thought I may be able to share some additional information.

Platform: Gitlab.com (17.3.0-pre) Renovate: Self-Hosted - v37.432.0 Git: v2.45.2 Manager: gomod

Project renovate.json

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "enabledManagers": ["gomod"],
  "postUpdateOptions": ["gomodUpdateImportPaths", "gomodTidy"],
  "packageRules": [{"excludePackageNames": ["go"]}]
}

Global config.js

module.exports = {
    endpoint: process.env.CI_API_V4_URL,
    repositories: [
        "company/project"
    ],
    customEnvVariables: {
        "GOPRIVATE": "gitlab.com/company/*",
        "GONOPROXY": "gitlab.com/company/*"
    },
    platform: 'gitlab',
    gitAuthor: 'Renovate Bot <renovate-bot@company.com>',
    autodiscover: false,
};

We have a private shared library of go modules with submodules inside of it. It's structure looks similar to:

gitlab.com/
  company/
    lib/           <<- repository root
      main.go
      go.mod
      so.sum
      sub-module/
        main.go
        go.mod
        go.sum

We maintain the library tags alongside the submode tags, so the releases for this repository look like:

 - v0.1.0 (lib release)
 - sub-module/v1.0.2 (sub-module release)
 - v0.0.12 (lib release)

Example go.mod

module gitlab.com/company/service

go 1.22.2

require (
    gitlab.com/company/lib v0.1.0
    gitlab.com/company/lib/sub-module v1.0.2
)

It appears that when Renovate looks up the projects from go mod - it correctly determines the root repository URL as https://gitlab.com/company/lib which allows it to correctly fetch the tags for the library via https://gitlab.com/api/v4/projects/company%2Flib/repository/tags.

But when it tries to lookup the sub-module information, it correctly grabs the root repository path https://gitlab.com./company/lib but then tries to lookup the tags using https://gitlab.com/api/v4/projects/company%2Flib%2Fsub-module/repository/tags. Which does not exist, as this assumes the repository root is located at https://gitlab.com/company/lib/sub-module.

I believe we'll be able to allow sub-module updates if we can fetch tags from the root repository, then filter by sub-module name prefix.

I hope this helps, Thanks for all your work

rarkins commented 2 months ago

How would your go.mod look like if you had a subgroup instead of a submodule? Eg if it were gitlab.com/company/group/subgroup-lib then how could Renovate determine it's a subgroup and not a submodule?

bassforce86 commented 2 months ago

In the go.mod file it wouldn't visually appear any different. The difference would be determined when making the ?go-get=1 calls to the referenced modules.

So for Gitlab at least, when the call is made to fetch the repository initially, it reports back the repository root.

so in the case of an actual subgroup, it would return the repository root as the subgroup. To follow the examples above it would appear something like this;

 gitlab.com/
  company/
    lib/   <- repository root return when calling `https://gitlab.com/company/lib?go-get=1` or when calling `https://gitlab.com/company/lib/sub-module?go-get=1`
      sub-module/ 
        main.go
        go.mod
        go.sum
      sub-group/ <-repository root returned when calling `https://gitlab.com/company/lib/sub-group?go-get=1`
        main.go
        go.mod
        go.sum

So we may be able to determine the repository to fetch tags from, based on the root repository returned from the initial go-get calls to either sub-groups, sub-modules, or regular repositories.

The only caveat for sub-modules is that we'll need to check for tag prefixes to correctly determine sub-module versions, this also will only apply to sub-modules that have had semantic versioned (and prefixed) tags applied, as the automated versioning applied by go & git does not prefix tags. This however is a project maintainer issue in my opinion, something worth documenting to help avoid bug reporting, but not something I would expect Renovate to handle.

zharinov commented 2 months ago

Now I'm starting to get why GitLab APIs require to url-escape the repository part in their APIs, for example the /projects/:id/repository/tags endpoint. Unlike GitHub, it allows their projects/repositories to easily have more than 2 segments e.g. foo/bar/baz. Is my understanding correct that repo names like that could be the case? For example, we already have some unit tests assuming it.

My suggestion is that we're providing the incorrect :id part to GitLab API when querying the tags, i.e. using foo/bar/baz where it must be foo/bar (and filtering of tags like baz/v1.0.0, baz/v2.0.0, etc is already implemented and should do the trick).

rarkins commented 2 months ago

Yes, they allow practically unlimited depth of subgroups

renovate-release commented 2 months ago

:tada: This issue has been resolved in version 38.18.7 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

zharinov commented 1 month ago

At this point we would like to have full collection of possible go module variants in GitLab:

I.e. be able to refer them from GitHub repository for testing purposes.

zharinov commented 1 month ago

...and how to install them using go tooling