search

renovatebot / renovate

Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io
https://mend.io/renovate
GNU Affero General Public License v3.0
17.41k stars 2.28k forks source link

Parsing problem with Gradle Version Catalog #14452

Closed keyute closed 1 year ago

keyute commented 2 years ago

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

31.94.0

Please select which platform you are using if self-hosting.

GitLab self-hosted

If you're self-hosting Renovate, tell us what version of the platform you run.

14.8.2-ee

Was this something which used to work for you, and then stopped?

I never saw this working

Describe the bug

Renovate looks up maven-metadata.xml at the wrong path when using library(alias, group, artifact).version(string)

Example library("exposed-core", "org.jetbrains.exposed", "exposed-core").versionRef("exposed") library("caffeine", "com.github.ben-manes.caffeine", "caffeine").version("3.0.4")

Relevant debug logs

Logs ``` Copy/paste any log here, between the starting and ending backticks {"name":"renovate","hostname":"***","pid":54,"level":20,"logContext":"***","repository":"***","url":"https://repo.maven.apache.org/maven2/exposed-core/org.jetbrains.exposed/maven-metadata.xml","msg":"Content is not found for Maven url","time":"2022-02-28T14:39:20.204Z","v":0} {"name":"renovate","hostname":"***","pid":54,"level":20,"logContext":"***","repository":"***","url":"https://repo.maven.apache.org/maven2/caffeine/com.github.ben-manes.caffeine/maven-metadata.xml","msg":"Content is not found for Maven url","time":"2022-02-28T14:39:18.968Z","v":0} ```

Have you created a minimal reproduction repository?

No reproduction repository

github-actions[bot] commented 2 years ago

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

keyute commented 2 years ago

Repository can be found here: https://github.com/keyute/renovate-issue

Confirmed reproducible via your hosted dashboard.

rarkins commented 2 years ago

Hi, you mention "Renovate looks up maven-metadata.xml at the wrong path" but could you provide more details about what you think is the right path?

Can you also include here the packageFiles with updates log and any relevant error logs from the reproduction repo?

keyute commented 2 years ago

With the above example, the correct path would be https://repo.maven.apache.org/maven2/org/jetbrains/exposed/exposed-core/maven-metadata.xml instead of https://repo.maven.apache.org/maven2/exposed-core/org.jetbrains.exposed/maven-metadata.xml

keyute commented 2 years ago

Here's the full log for the minimal repository I've included which confirms the issue

Logs ``` DEBUG: No dangling containers to remove INFO: Repository started { "renovateVersion": "31.93.0" } DEBUG: Using localDir: /mnt/renovate/gh/keyute/renovate-issue DEBUG: Repository cache is valid DEBUG: initRepo("keyute/renovate-issue") DEBUG: Overriding default GitHub endpoint { "endpoint": "https://api.github.com/" } DEBUG: keyute/renovate-issue default branch = main DEBUG: Using app token for git init DEBUG: resetMemCache() DEBUG: Resetting npmrc DEBUG: detectSemanticCommits() DEBUG: checkOnboarding() DEBUG: isOnboarded() DEBUG: Checking cached config file name DEBUG: Existing config file confirmed DEBUG: Repo is onboarded DEBUG: migrateAndValidate() DEBUG: No config migration necessary DEBUG: massaged config { "config": { "extends": [ "github>whitesource/merge-confidence:beta", "config:base" ] } } DEBUG: migrated config { "config": { "extends": [ "github>whitesource/merge-confidence:beta", "config:base" ] } } DEBUG: Setting hostRules from config DEBUG: Found repo ignorePaths { "ignorePaths": [ "**/node_modules/**", "**/bower_components/**", "**/vendor/**", "**/examples/**", "**/__tests__/**", "**/test/**", "**/tests/**", "**/__fixtures__/**" ] } DEBUG: No vulnerability alerts found DEBUG: No vulnerability alerts found DEBUG: findIssue(Dependency Dashboard) DEBUG: No baseBranches DEBUG: extract() DEBUG: Found cached extract { "baseBranch": "main", "baseBranchSha": "eb3afd7ca9bf6ac86fd450928ebca0215d881529" } DEBUG: Deleted cached dep updates INFO: Dependency extraction complete { "baseBranch": "main", "stats": { "managers": { "gradle": { "fileCount": 3, "depCount": 2 }, "gradle-wrapper": { "fileCount": 1, "depCount": 1 } }, "total": { "fileCount": 4, "depCount": 3 } } } DEBUG: Looking up caffeine:com.github.ben-manes.caffeine in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/org/jetbrains/kotlin/jvm/org.jetbrains.kotlin.jvm.gradle.plugin/maven-metadata.xml" } DEBUG: Looking up org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 85 new releases for org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/caffeine/com.github.ben-manes.caffeine/maven-metadata.xml" } DEBUG: Failed to look up dependency caffeine:com.github.ben-manes.caffeine (caffeine:com.github.ben-manes.caffeine)(packageFile="settings.gradle.kts", dependency="caffeine:com.github.ben-manes.caffeine") DEBUG: Package releases lookups complete { "baseBranch": "main" } DEBUG: branchifyUpgrades DEBUG: 0 flattened updates found: DEBUG: Returning 0 branch(es) DEBUG: config.repoIsOnboarded=true DEBUG: packageFiles with updates { "config": { "gradle": [ { "packageFile": "gradle.properties", "datasource": "maven", "deps": [] }, { "packageFile": "settings.gradle.kts", "datasource": "maven", "deps": [ { "depName": "caffeine:com.github.ben-manes.caffeine", "currentValue": "caffeine", "managerData": { "fileReplacePosition": 183, "packageFile": "settings.gradle.kts" }, "fileReplacePosition": 183, "registryUrls": [ "https://repo.maven.apache.org/maven2" ], "depIndex": 0, "warnings": [ { "topic": "caffeine:com.github.ben-manes.caffeine", "message": "Failed to look up dependency caffeine:com.github.ben-manes.caffeine" } ], "versioning": "gradle", "updates": [] } ] }, { "packageFile": "build.gradle.kts", "datasource": "maven", "deps": [ { "depType": "plugin", "depName": "org.jetbrains.kotlin.jvm", "lookupName": "org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin", "registryUrls": [ "https://repo.maven.apache.org/maven2", "https://plugins.gradle.org/m2/" ], "commitMessageTopic": "plugin org.jetbrains.kotlin.jvm", "currentValue": "1.6.10", "managerData": { "fileReplacePosition": 37, "packageFile": "build.gradle.kts" }, "fileReplacePosition": 37, "depIndex": 0, "warnings": [], "versioning": "gradle", "currentVersion": "1.6.10", "fixedVersion": "1.6.10", "updates": [] } ] } ], "gradle-wrapper": [ { "packageFile": "gradle/wrapper/gradle-wrapper.properties", "deps": [ { "depName": "gradle", "currentValue": "7.4", "replaceString": "https\\://services.gradle.org/distributions/gradle-7.4-bin.zip", "datasource": "gradle-version", "versioning": "gradle", "depIndex": 0, "warnings": [], "sourceUrl": "https://github.com/gradle/gradle", "homepage": "https://gradle.org", "currentVersion": "7.4", "fixedVersion": "7.4", "updates": [] } ] } ] } } DEBUG: processRepo() DEBUG: Processing 0 branches: DEBUG: Calculating hourly PRs remaining DEBUG: Retrieving PR list DEBUG: Retrieved 1 Pull Requests DEBUG: currentHourStart=2022-03-01T09:00:00.000+00:00 DEBUG: PR hourly limit remaining: 2 DEBUG: Calculating prConcurrentLimit (10) DEBUG: 0 PRs are currently open DEBUG: PR concurrent limit remaining: 10 DEBUG: Calculated maximum PRs remaining this run { "prsRemaining": 2 } DEBUG: PullRequests limit = 2 DEBUG: Calculating hourly PRs remaining DEBUG: currentHourStart=2022-03-01T09:00:00.000+00:00 DEBUG: PR hourly limit remaining: 2 DEBUG: Calculating branchConcurrentLimit (10) DEBUG: 0 already existing branches found: DEBUG: Branch concurrent limit remaining: 10 DEBUG: Calculated maximum branches remaining this run { "branchesRemaining": 2 } DEBUG: Branches limit = 2 DEBUG: Ensuring Dependency Dashboard DEBUG: ensureIssue(Dependency Dashboard) INFO: Cannot ensure issue because issues are disabled in this repository DEBUG: Removing any stale branches DEBUG: config.repoIsOnboarded=true DEBUG: No renovate branches found INFO: Cannot ensure issue because issues are disabled in this repository DEBUG: Repository timing splits (milliseconds) { "splits": { "init": 2277, "extract": 2, "lookup": 871, "update": 238 }, "total": 3410 } DEBUG: http statistics { "urls": { "https://api.github.com/graphql (POST)": 2, "https://api.github.com/repos/keyute/renovate-issue/contents/renovate.json (GET)": 1, "https://api.github.com/repos/keyute/renovate-issue/pulls (GET)": 1, "https://api.github.com/repos/whitesource/merge-confidence/contents/beta.json (GET)": 1, "https://plugins.gradle.org/m2/org/jetbrains/kotlin/jvm/org.jetbrains.kotlin.jvm.gradle.plugin/1.6.10/org.jetbrains.kotlin.jvm.gradle.plugin-1.6.10.pom (GET)": 1 }, "hostStats": { "api.github.com": { "requestCount": 5, "requestAvgMs": 215, "queueAvgMs": 0 }, "plugins.gradle.org": { "requestCount": 1, "requestAvgMs": 121, "queueAvgMs": 4 } }, "totalRequests": 6 } INFO: Repository finished { "durationMs": 3410 } ```
PhilipAbed commented 2 years ago

Hi @keyute i'm trying to reproduce but it's strange as i'm getting different logs and an extra 2 PRs that you don't have on your repository.

i'm getting :

DEBUG: Found 4 package file(s) (repository=StinkyLord/parsing-gradle-maven-url)
 INFO: Dependency extraction complete (repository=StinkyLord/parsing-gradle-maven-url)
       "baseBranch": "main",
       "stats": {
         "managers": {
           "gradle": {"fileCount": 3, "depCount": 1},
           "gradle-wrapper": {"fileCount": 1, "depCount": 1}
         },
         "total": {"fileCount": 4, "depCount": 2}
       }
DEBUG: Looking up org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://repo.maven.apache.org/maven2/ (repository=StinkyLord/parsing-gradle-maven-url)
DEBUG: Found 3 new releases for org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://repo.maven.apache.org/maven2/ (repository=StinkyLord/parsing-gradle-maven-url)
DEBUG: Looking up org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://plugins.gradle.org/m2/ (repository=StinkyLord/parsing-gradle-maven-url)
DEBUG: Found 91 new releases for org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://plugins.gradle.org/m2/ (repository=StinkyLord/parsing-gradle-maven-url)
DEBUG: PackageFiles.add() - Package file saved for branch (repository=StinkyLord/parsing-gradle-maven-url)

Notice the number of dependencies found is 2 not 3,

and in the config :

 "config": {
         "gradle": [
           {"packageFile": "gradle.properties", "datasource": "maven", "deps": []},
           {"packageFile": "settings.gradle.kts", "datasource": "maven", "deps": []},
           {
             "packageFile": "build.gradle.kts",
             "datasource": "maven",
             "deps": [
               {
                 "depType": "plugin",
                 "depName": "org.jetbrains.kotlin.jvm",
                 "packageName": "org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin",
                 "registryUrls": [
                   "https://repo.maven.apache.org/maven2",
                   "https://plugins.gradle.org/m2/"
                 ],

Notice that there is no 

  "packageFile": "settings.gradle.kts",
        "datasource": "maven",
        "deps": [

in my logs.

I tried reproducing on windows cli + docker + renovate app, I wasn't able to reproduce, there's no mention of caffeine in my logs!

PhilipAbed commented 2 years ago

i was able to reproduce on an older version of renovate through docker

DEBUG: Found gradle-wrapper package files (repository=StinkyLord/parsing-gradle-maven-url)
DEBUG: Found 4 package file(s) (repository=StinkyLord/parsing-gradle-maven-url)
 INFO: Dependency extraction complete (repository=StinkyLord/parsing-gradle-maven-url)
       "baseBranch": "main",
       "stats": {
         "managers": {
           "gradle": {"fileCount": 3, "depCount": 2},
           "gradle-wrapper": {"fileCount": 1, "depCount": 1}
         },
         "total": {"fileCount": 4, "depCount": 3}
       }
DEBUG: Looking up caffeine:com.github.ben-manes.caffeine in repository https://repo.maven.apache.org/maven2/ (repository=StinkyLord/parsing-gradle-maven-url)
DEBUG: Looking up org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://repo.maven.apache.org/maven2/ (repository=StinkyLord/parsing-gradle-maven-url)
DEBUG: Content is not found for Maven url (repository=StinkyLord/parsing-gradle-maven-url)
       "url": "https://repo.maven.apache.org/maven2/caffeine/com.github.ben-manes.caffeine/maven-metadata.xml",
       "statusCode": undefined

reproduced on version 32.10.0, we might have a bigger issue as this is not reproducible on latest versions of renovate, but ignored!

PhilipAbed commented 2 years ago

This format

library(alias, group, artifact).version(string)

is not supported in renovate i think this is a feature request

Churro commented 2 years ago

@PhilipAbed, your analysis is correct. While testing #15903 with a different repo, I also stumbled upon this issue. I already have a fix for it ready but wanted to wait for the other PR to be merged to prevent merge conflicts.

PhilipAbed commented 2 years ago

@keyute i can see that @Churro already merged https://github.com/renovatebot/renovate/pull/15929 can you try to check if it is fixed so we can close this issue?

Churro commented 1 year ago

Closing as this has been resolved with https://github.com/renovatebot/renovate/pull/15929.

The test repo https://github.com/keyute/renovate-issue is no longer available but tests confirm that the initially mentioned problem cases are meanwhile working: https://github.com/renovatebot/renovate/blob/cc50beb0934874095fd2164b33dcb5fed7dbf421/lib/modules/manager/gradle/parser.spec.ts#L98-L99