Add support for Gradle Version Catalogs (Gradle 7.0)

[ReginFell]

Gradle 7.0 was released recently and it introduces a new Central Declaration System as well as a new supported format .toml. https://github.com/gradle/gradle/issues/15352

Would be nice if Dependabot supported this.

[JavierSegoviaCordoba]

I would like that dependabot search for ALL toml files in the project (regardless of the directory or subdirectory the file is in).

[madhead]

As an expansion for this…

I was using a trick with a Versions.kt file in buildSrc hosting all the versions for my project and using it in the build. Take a look at the example in madhead/ImgMacroBot:

buildSrc/src/main/kotlin/Versions.kt

object Versions {
    const val JVM = "11"

    object Plugins {
        const val KOTLIN = "1.4.21"
        const val DETEKT = "1.15.0"
        const val LIQUIBASE = "2.0.4"
    }

    object Dependencies {
        const val KTOR = "1.5.1"
        const val KOIN = "2.2.2"
        const val KOTLINX_SERIALIZATION = "1.0.1"
        const val TGBOTAPI = "0.30.0"
        const val SKIJA = "0.89.18"
        const val MICROMETER = "1.6.4"
        const val LOG4J = "2.14.0"
        const val JACKSON = "2.12.1"
        const val JUNIT = "5.7.1"
        const val MOCKK = "1.10.6"
        const val LIQUIBASE = "4.3.1"
        const val SNAKEYAML = "1.28"
        const val OJDBC11 = "21.1.0.0"
    }
}

And then in any build.gradle.kts:

dependencies {
    implementation(platform("org.apache.logging.log4j:log4j-bom:${Versions.Dependencies.LOG4J}"))
    implementation(platform("io.micrometer:micrometer-bom:${Versions.Dependencies.MICROMETER}"))

    api("io.micrometer:micrometer-core")
    api("dev.inmo:tgbotapi:${Versions.Dependencies.TGBOTAPI}")
    api("org.jetbrains.skija:skija-linux:${Versions.Dependencies.SKIJA}")
    …
}

Dependabot seems not to handle that correctly, check the repo's insights: the dependencies list is empty. I don't know if Dependabot actually executes Gradle to parse the dependencies, but seems like it's abilities in parsing them are limited.

Probably, if it executed full-featured Gradle it could handle any scenarios like mine or like the new central declaration system.

[huehnerlady]

@madhead that sounds like a completely different issue. Here we ask for the support of the new gradle feature with toml files and not for the buildSrc workaround. Maybe it makes sense to open a separate issue for that?

[JavierSegoviaCordoba]

Another thing, a Version Catalog can be published, and you can add them to whatever project you want via settings.gradle files:

dependencyResolutionManagement {
    repositories {
        mavenCentral()
        google()
        gradlePluginPortal()
    }

    versionCatalogs {
        create("libs") { from("com.example.catalogs:some-catalog:1.0.0") }
    }
}

So should be great dependabot can search for updates for them.

[madhead]

Take it easier!

My point is not about that particular buildSrc example, it's just an example. I'm trying to add a comment that if only Dependabot actually evaluate the build script and extract the versions from the model, it could support all the scenarios, including this new TOML feature and aforementioned versionCatalogs.

P.S. I don't know how does Dependabot work currently, but it seems like it uses a bunch of regexps to parse the scripts. I understand it could be the only option - evaluating a build scripts needs a Java, which might be unacceptable. It's also impossible to make a PR based on the evaluated model, but it's just an idea.

[tbroyer]

@madhead There's #1164 already. It wouldn't tell Dependabot which file to update though, for automatic pull requests.

[madhead]

@tbroyer, yep, looks exactly what I was talking about! And, yeah, updates are almost impossible with this approach.

[JavierSegoviaCordoba]

Anyway, having dependencies in buildSrc is not only a bad practice, there are unlimited ways to organize them there, so it is really difficult to cover all use cases.

Knowing version catalogs will be the standard way for dependency management, I think there is no sense in searching a way to get it working in buildSrc.

[DanySK]

There's a bit of confusion. There are two different issues at hand:

1. supporting the central declaration system;
2. supporting Gradle fully.

Although 2. (see #1164 as @tbroyer suggests) would likely cover both, adding 1. first would be useful in many cases and easier.

[ReginFell]

Yeah, parsing toml should be relatively simple task since dependabot already has this for Python

[huehnerlady]

I would say the ultimate goal is to support gradle fully. if this could be done in the near future, then no need for the central dependencies. BUT as it does not seem to be near I would say it might be a very good way to migrate towards more support of gradle-kotlin-dsl :)

[huehnerlady]

Is there any update on this? Would be great to have this feature and I thought someone mentioned it should not be tooo much work?

[jurre]

It's not currently on our roadmap, I'm not sure who said that, but it seems like a significant amount of work from a quick glance

[DanySK]

@jurre, the comparison was being made with the proposal for going through the Gradle Tooling API #1164; my feeling is that the amount of work required to support the central declaration in a TOML file would be way smaller...

[jurre]

@jurre, the comparison was being made with the proposal for going through the Gradle Tooling API #1164; my feeling is that the amount of work required to support the central declaration in a TOML file would be way smaller...

Yes, I agree that it seems smaller, but still a significant amount of work that we currently don't have planned, so I can't give a timeline

[DanySK]

I see; thanks for the clarification.

[huehnerlady]

@jurre thanks for the clarification. is there anything we can do to get it on the roadmap asap? 😉

[be1ski]

@jurre we are very sad to hear that 🥲 maybe you have at least an approximate date for the appearance of this feature?

[StefanOltmann]

Sad. Dependabot does not work with Gradle in most use cases.

[jurre]

@jurre we are very sad to hear that 🥲 maybe you have at least an approximate date for the appearance of this feature?

@asciimike would be the best person to help answer that, but I don't think there is much we can commit to at this point

[DanySK]

@StefanOltmann @be1ski one way around the current level of support is to manage dependencies via refreshVersions and automatic upgrades via UpGradle. I'm currently achieving dependabot-like functionality this way, while I wait for support to Gradle to be complete.

[JavierSegoviaCordoba]

I am being forced to generate the TOML files from code inside build.gradle.kts files to get dependabot working.

[Nambers]

Is there any update on this? Would be great to have this kind of feature like supporting the toml or buildSrc in gradle. But my dependabot look like dont get my toml file.

[asciimike]

No updates on timeline, as we don't have the gradle expertise or time at present to properly prioritize this (or #1164, which would be even better). If folks are interested in working on this, let us know.

[dmedinag]

I have to add myself to the interested parties in seeing this happen. TOML catalogs introduce a massive leap forward in dependency management for frameworks developed with gradle and since we're using my team and I have had to say goodbye to Dependabot :( We'd love to have it back in our project.

[DanySK]

@dmedinag as soon as the refreshVersions plugin will support TOML catalogs, UpGradle will provide Dependabot-like functions for Gradle projects through refreshVersions.

[tbroyer]

@DanySK Dependabot powers GitHub's security alerts, it's not only about having automatic updates through pull requests.

[DanySK]

@tbroyer sure, but if @dmedinag is dropping all of Dependabot functionality, in that way at least he can get updates back. Having dependabot supporting Gradle better would be great for me too (that's why I'm following the discussion here...).

[jurre]

@DanySK Dependabot powers GitHub's security alerts, it's not only about having automatic updates through pull requests.

These are currently actually separate systems, Alerts power security updates, but dependabot is not used to scan for alerts at this time. (We hope to bring those systems implementations together at some point, but we've not been able to prioritize it)

[tbroyer]

@jurre As someone who would like alerts, and doesn't really care about automatic updates, is the code open source and could I contribute to it to add support for those TOML files?

[jurre]

@jurre As someone who would like alerts, and doesn't really care about automatic updates, is the code open source and could I contribute to it to add support for those TOML files?

Nope, unfortunately all of that is private :(

I'll check with the team to see if it's on their radar though

[tbroyer]

And for Alerts, lockfiles or verification metadata would be great (and even better, when used)

[plastiv]

For people who stumbled across this issue I would like to let you know that Renovate Bot supports Gradle Versions Catalogs as of Sep 2021 https://github.com/renovatebot/renovate/issues/8692#issuecomment-911897526

I hope I don't violate TOC by advertising competitors 😄 .

[rr-nick-tan]

hi dependabot team, do you have any plan to have it support this year 2022? otherwise, i will have to switch to renovatebot.

[kimble]

Version catalogs are becoming stable in Gradle 7.4 https://docs.gradle.org/7.4-rc-1/release-notes.html

[TWiStErRob]

@jurre can you please comment on whether this is going to be supported? Gradle 7.4 is out, the feature is stable.

• [ ] I think at the bare minimum the conventional default libs.versions.toml could be supported
  as mentioned by @ReginFell toml support is there in python,
• [ ] and then multiple/different file names and locations as comment 2 put it
• [ ] then maybe settings.gradle(.kts) definition (hardcoded),
• [ ] then maybe settings.gradle(.kts) definition (imported) as separate future improvements?

It doesn't have to be full-on immediate support, but Gradle is so widely used that even the slightest support would help a lot of projects, right?

[GrahamBorland]

FWIW I recommend Renovate which works beautifully with version catalogs. (I have no affiliation, just a happy user.)

[gesellix]

What about GitHub/Dependabot working together with Renovate, combining code bases (or create a shared lib for both tools) and ultimately making Dependabot recognize Gradle's Version Catalogs?

We all probably don't want or even need the perfect solution, which might be impossible given Gradle's rich dependency model and feature set.

[JavierSegoviaCordoba]

That implies GitHub or Microsoft buying Renovate company (supposing Renovate company wants to allow that)

[TWiStErRob]

Yeah Mend is a competitor to GitHub on many fields. On the competitor side, acquisition seems unlikely, because Renovate was just acquired a few years ago, and have been rebranded and integrated recently.

But in an ideal world the two companies would work together to create something better for the users, or at least compete 🤔. As it currently stands, GitHub's solution discourages/blocks progress in the Java/Kotlin ecosystem. 🥺

[ghost]

Yeah Mend is a competitor to GitHub on many fields. On the competitor side, acquisition seems unlikely, because Renovate was just acquired a few years ago, and have been rebranded and integrated recently.

But in an ideal world the two companies would work together to create something better for the users, or at least compete 🤔. As it currently stands, GitHub's solution discourages/blocks progress in the Java/Kotlin ecosystem. 🥺

Microsoft just hates Java because they know that it's better than C#

[jurre]

@jurre can you please comment on whether this is going to be supported? Gradle 7.4 is out, the feature is stable.

Hey, apologies for the late response, I've been out on parental leave for the last few months.

Yes, it's something that we want to support, but I can't speak to the timeline unfortunately. Is there anything you can point to to gauge adoption within the community? It would help a lot with prioritization.

[colintheshots-meetup]

I can't speak for the entire community or every production project, but here at Meetup we've had a new app in production for more than a year now using Gradle version catalogs exclusively. We're never going back to the old ways.

[PaulWoitaschek]

The same at Yazio. I don't think there is a way to pull statistical data here.

However: Version catalogs is the standard way to go. Afaik this is the one single recommendation that Gradle gives on how to manage dependency Versions.

So now that it's no longer incubating I expect the adoption to only increase. And every project that migrates to Version catalogs will probably also migrate from dependabot to renovate.

[TWiStErRob]

Every Gradle project I touch I migrate to Version Catalogs as they solve the many issues of the old workarounds and provide a much better support foundation for tooling.

Re community usage: Opted in: filename:libs.versions.toml out of potential adopters: https://github.com/search?q=%22gradle-7.4-bin.zip%22&type=code (this query needs work, not sure why it's not working).

Once the above query works, you should get a full list of support from "gradle-7.4-bin.zip" OR "gradle-7.4-all.zip" OR "gradle-7.4.1-bin.zip" OR "gradle-7.4.1-all.zip" OR "gradle-7.4.2-bin.zip" OR "gradle-7.4.2-all.zip" OR "gradle-7.5-bin.zip" OR "gradle-7.5-all.zip"

[Thunderforge]

For what it's worth, the Fortune 500 company I work for wants to use Version Catalogs, but doesn't want to give up Dependabot.

So it may be a Catch-22: if Dependabot won't support it until it's more popular, it won't be more popular until Dependabot supports it.

[JakeWharton]

There's a way to pull statistical data but it's GitHub's responsibility to do so, not the community. They're literally sitting on the largest corpus of software projects both public and private. They have the ability to ask things like "How many projects updated in the last 6 months with a gradlew file also have a gradle/libs.versions.toml" with ease. Or better yet, plot a trend line of adoption. I don't think a bunch of "I use it, too" comments offer anything statistically significant or relevant.

Btw: I use it, too.

[eygraber]

I don't know if Gradle has any telemetry data on it; I'm assuming that would be useful if they did.

I've been using version catalogs since they were in preview, and I've been pushing for their adoption to anyone who will listen.

I know @ben-manes has been suggesting that someone (Gradle) assume responsibility for his plugin. Maybe something can be done with that?

[alwaysbemark]

Gradle version catalogues have been truly life-changing for us, at this stage I would never use anything but for new JVM projects. We use them within settings.gradle files under dependencyResolutionManagement rather thanin .toml files but if one is easier to deliver than the other - we're happy to switch.

[ben-manes]

Since the GitHub Dependency Graph api came out in beta, I think the next step is for @JLLeitschuh to deploy a prototype of Gradle's github-dependency-extractor plugin. That would avoid many of the headaches, but unfortunately there has not been much recent communication around this topic.