Add support for Gradle Version Catalogs (Gradle 7.0)

[ReginFell]

Gradle 7.0 was released recently and it introduces a new Central Declaration System as well as a new supported format .toml. https://github.com/gradle/gradle/issues/15352

Would be nice if Dependabot supported this.

[rciovati]

While I agree with @JakeWharton that it's GitHub's responsibility to obtain and track this data, I ran a couple low-effort and low-confidence queries with Sourcegraph and these are the results:

• 5k+ open source projects reference Gradle 7.4 or 7.5, not including forks (source)
• ~700 repos have a libs.versions.toml file (source)

These are indeed lower limits as Sourcegraph includes only open source projects

[jurre]

Thanks @rciovati, of course I am happy to perform a query on our end that includes private repo counts, but I am not entirely sure what the right query would be to get a good idea of what adoption looks like. I see libs.versions.toml mentioned, but I also see dependencyResolutionManagement in settings.gradle mentioned, should these both be included?

As for the 7.4 or 7.5 version, this does not automatically mean that these projects are using version catalogs right?

I'm not super familiar with gradle so I'm just trying to make sure that I paint a fair picture when I try to get this prioritized 🤷

[JLLeitschuh]

Since the GitHub Dependency Graph api came out in beta, I think the next step is for @JLLeitschuh to deploy a prototype of Gradle's github-dependency-extractor plugin. That would avoid many of the headaches, but unfortunately there has not been much recent communication around this topic.

The reason there's not been much communications around this is that I was the one at Gradle spearheading this effort and I left Gradle last January for the Dan Kaminsky Fellowship. I'd personally love to finish this story (or find a company willing to fincance finishing this story) but I'm not personally available until the end of the year.

At this point, the best next step would be for us to get some beta testers for the Gradle Plugin I wrote that pulls the dependency information out of an executing Gradle Build and sends that information to GitHub.

Anyone who is willing to invest time into this and genuinely is willing to contribute and wants to setup a meeting with me to discuss finishing this project, find the link below to my calendly. NOTE: I'm in a bit of crunch time attempting to finish my work that I'm presenting at Black Hat, and DEFCON, so I won't be available until after the second week of August. But if anyone wants to chat after then, feel free to put something on my calendar.

https://calendly.com/jonathan-leitschuh-at-dan-kaminsky-fellowship/30min

[mikepenz]

As an alternative approach to the above, I've been experimenting with a GitHub action, which would parse the output returned from the Gradle dependencies task, making it theoretically work for any gradle project (and it would not require other adoptions to the project).

You can look at the code here: https://github.com/mikepenz/gradle-dependency-submission (https://github.com/mikepenz/gradle-dependency-submission/blob/main/src/process.ts#L104-L110) While it works, and would be project independent, there are surely better alternatives, as highlighted above.

And the dependencies highlighted within the Dependency Graph

The problem is that right now most maven packages are not linked to the repositories (even with the pom.xml defining the right url), and even worse some dependencies linking to wrong repositories. Reported here: https://github.com/community/community/discussions/19492

A second problem seems to be that the dependency graph for maven packages ignores the hierarchy of dependencies and will only render a flat list. Reported here: https://github.com/community/community/discussions/19495

[rciovati]

I see libs.versions.toml mentioned, but I also see dependencyResolutionManagement in settings.gradle mentioned, should these both be included?

@jurre Yes, if you want more precise result you should also include any generic *.versions.toml file and the other ways of using the feature that are mentioned in the doc. Declaring dependencies in libs.versions.toml is just the simplest way of using Gradle's version catalogs. Arguably that would already be very useful for many people.

As for the 7.4 or 7.5 version, this does not automatically mean that these projects are using version catalogs right?

Correct, that only means that they have a Gradle version that support version catalogs.

[bric3]

@jurre Version catalog and dependencyResolutionManagement are not mutually exclusive, I think you can count any project that has at least a gradle/*.versions.toml, maybe gradle/*.toml.

And yes you can count me as well, for public repositories, as well as private ones in my orgs (project created/maintained by other teams).

[jmfayard]

I think the issue can be marked as resolved, judging by https://github.com/square/okhttp/pull/7386

EDIT: stupid mistake, sorry

[martinbonnin]

@jmfayard that okhttp issue is made by Renovate. This is the dependabot repo.

[LachlanMcKee]

@jmfayard I believe the main reason that having dependabot work with Gradle versions catalogue is because it has deeper integration with Github

[huehnerlady]

Is there any update on this? Unfortunately this limits our use of dependabot massively as we are using toml files in most of our repositories :(

[ben-manes]

@huehnerlady I use gradle-dependency-submission which works pretty well. While github is confused about where the dependency is defined (so no PRs), it does know the coordinates to provide update and security alerts. I still use gradle-versions-plugin primarily (which is toml compatible) and wanted to leverage github’s security features for a public repository.

[TWiStErRob]

Wow, workaround^2 :) Man, you need to experience fully-automatic PR-based CI-protected version bumps. Once you go there, there's no way back to accepting anything less. Thanks for highlighting the dependency submission API though, that's a nice feature for security alerts.

I'd like to call out to Members, to reiterate that we're not expecting a full-on runtime-gathered fully-full Gradle Version Catalog support, any minimal support (even regex parsing) would be a huge step forward (https://github.com/dependabot/dependabot-core/issues/3121#issuecomment-1077886075).

[alsutton]

@jurre Any chance you could take a look at https://github.com/dependabot/dependabot-core/pull/6249 with a view to approving it which would provide a solution to this issue?

(I can see from the comments you helped the folk develop it)

[warting]

Today is the day my dreams came true! I noticed a few new PR from dependabot in my projects today bumping various kind of libraries in my toml files! :heart:

[planetf1]

@warting so dependabot now appears to support toml files now? if so yay indeed!

[deivid-rodriguez]

As you observed, we added Gradle Version Catalog support 🎉.

Thanks for your patience! Please report any issues or feature requests over the initial MVP as separate issues :)

[svenjacobs]

The documentation doesn't seem to be updated yet. Do I need to specify directory: "/gradle" in dependabot.yml or how does Dependabot distinguish between build.gradle, build.gradle.kts and gradle/libs.version.toml?

[huehnerlady]

I am also looking for a documentation what the MVP is actually supporting in regards to toml files. As @svenjacobs says are there special requirements that needs to be met in order to get it to work?

And are only versions supported, or also plugins and libraries for example, documented here?

[alsutton]

For those who want some docs, we've just deployed dependabot on our version catalogue for Gradle and we're getting update PRs. The only thing we needed to do was create /.github/dependabot.yml in our repo with the following contents;

version: 2
updates:
  - package-ecosystem: "gradle"
    directory: "/"
    schedule:
      interval: "daily"

which gave us auto-created PRs such as this one;

[deivid-rodriguez]

Hei, sorry for the lack of documentation, we have some updates there but we haven't been able to deploy them yet.

This initial version supports updating the "standard" version catalog file gradle/libs.versions.toml, and it supports updating versions, plugins and libraries. Other setups configuring custom catalogs at settings.gradle are not yet supported.

[huehnerlady]

@deivid-rodriguez thanks for the update. so there will be a documentation with examples? We are using "versions" and nothing more and wonder if that is not yet supported. Could you maybe verify?

our libs.versions.toml:

[versions]
[...]
springBoot = "2.7.9"

and we are implementing it in the dependencies section of our built.gradle.kts like this:

dependencies {

[...]
  implementation("org.springframework.boot:spring-boot-starter-web:${libs.versions.springBoot.get()}")
  implementation("org.springframework.boot:spring-boot-starter-actuator:${libs.versions.springBoot.get()}")
[...]
}

[deivid-rodriguez]

No problem!

No, I don't think that's supported either, sorry. We support referencing versions as version.ref references in plugins or libraries inside the lib.versions.toml file. But I don't think accessing versions like that is supported.

The underlying issue is that we don't run Gradle/Java under the hood, so anything that requires evaluating Java code is not really supported at the moment.

[deivid-rodriguez]

Docs have also been updated now to mention the scope of the current support: https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file (footnote number 3 in package-ecosystem table).

[tjuszczyk]

This initial version supports updating the "standard" version catalog file gradle/libs.versions.toml, and it supports updating versions, plugins and libraries. Other setups configuring custom catalogs at settings.gradle are not yet supported.

@deivid-rodriguez Is there any. scheduled plan to support additional gradle/*.toml files as well as a default one?

[deivid-rodriguez]

Not right now. Our long term plan is to move on to using the Gradle Tooling API (#1164), so that we don't have to parse Java/Kotlin/Groovy with Ruby code, but that work is non trivial and hasn't been prioritized yet.

[tjuszczyk]

With gradle/libs.versions.toml supported, wouldn't supporting list of gradle/*.toml be simple addition?

[jeffwidman]

As Deivid said, for the core team we aren't planning to do a lot of further feature enhancements to the ruby/regex code because the better long term play is #1164.

However, this is open source, and we certainly welcome PR's if you want to take a stab at it! If you do start on it and run into questions let us know... we're always happy to answer questions from contributors... typically easiest to start with a draft PR and add comments where you get stuck.

[deivid-rodriguez]

It shouldn't be too hard, but we still need to parse the configuration in settings.gradle, which is written if Groovy. Given the dynamic nature of it, there's probably a few different ways to configure it, so it might get tricky, but we'd gladly review a PR implementing it. We already parse settings.gradle from Ruby for other things.

[Kernald]

It shouldn't be too hard, but we still need to parse the configuration in settings.gradle, which is written if Groovy. Given the dynamic nature of it, there's probably a few different ways to configure it, so it might get tricky, but we'd gladly review a PR implementing it. We already parse settings.gradle from Ruby for other things.

Note that this file might be written in Kotlin as well - our repo has a settings.gradle.kts containing a version catalog. I guess the next step for us is to migrate that one to a toml until #1164.