Implement JS-based Gradle manager

[rarkins]

What would you like Renovate to be able to do?

Support a JS-based parsing approach for Gradle, as aligned as possible to other managers (e.g. npm, Maven).

Describe the solution you'd like

Extract and update functions are JS-based, similar to Maven's.

Describe alternatives you've considered

Continuing to extend the existing plugin-based Gradle child process approach.

Additional context

I realised that there should be nothing wrong with having two Gradle managers, e.g. gradle and gradle-simple. It should even be fine to have both running at once, although probably we won't enable both by default long term.

So I'd like to explore how far we can get with a pure JS-based solution. I think there will obviously be some ways of defining dependencies that we won't support because it involves evaluation of Groovy logic, but I'm not sure anyone will need that approach. Either way, the existing gradle manager will still exist indefinitely.

Also, with this we aim to support Kotlin syntax and subdirectory-based projects too.

Cc @corecanarias @ChristianMurphy @bagage @jcornaz @jnizet @jansauer

[rarkins]

Files to match

Most important are build.gradle and build.gradle.kts. We should look for them anywhere they exist in the repo. These should be in the default fileMatch for the new manager.

We should also look for any gradle.properties or gradle.properties.kts files next to it, in case the gradle build files reference variables within.

What about settings.gradle?

Are there any other files in a Gradle project that we will need to look at? e.g. is there also the need to dynamically detect and resolve references? For the JS-based parser I'd hope to avoid this and say we only support dependency declarations in the build.gradle or gradle.properties.

[rarkins]

Gradle References

Writing build scripts: https://docs.gradle.org/current/userguide/writing_build_scripts.html DSL: https://docs.gradle.org/current/dsl/index.html Dependency handler DSL: https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.dsl.DependencyHandler.html Multiproject builds: https://docs.gradle.org/current/userguide/multi_project_builds.html#sec:dependencies_which_dependencies Declaring dependencies: https://docs.gradle.org/current/userguide/declaring_dependencies.html Dependency types: https://docs.gradle.org/current/userguide/dependency_types.html#dependency_types

Useful reference projects

https://github.com/ChristianMurphy/uPortal

more reference projects needed

[rarkins]

Content we care about

Ultimately, all dependency declarations will occur within a dependencies block. But this block can itself be within other blocks like buildscript, allprojects and subprojects.

Dependency versions can be defined right within the dependencies block, or indirectly via variables that are in the properties file.

We should also ideally handle Plugins too.

We probably also need to support configuration of non-default Maven registries, although Renovate's current approach would allow someone to configure that via Renovate config instead. But ideally if the user has configured registry details in their Gradle file(s) then we should avoid making them configure it again in their Renovate config.

[corecanarias]

I think the week of 29 Apr I will be able to work on this

[corecanarias]

If there is anybody else interested in collaborate, that would be great ;)

[rarkins]

@corecanarias I'm interested :)

BTW I have also been looking into whether we should use a "proper" parser like peg.js for the Gradle/Kotlin files rather than our usual regex-based approach. If a parser is a good idea, I might try to find an expert in that to kickstart it.

But alternatively.. I think the nice thing here is that we don't need to have a "full replacement" for Gradle right away, because we keep the existing Gradle manager. So we can start with the simplest example (dependencies within the buildscript without variables) and then keep building out from there. If we later decide that a proper parser is necessary for further expansion, we can abstract the parsing part so that it can "swap in" and reuse existing tests.

[rarkins]

Private registries

By default we should fail gracefully to look up dependencies from non-maven.org, but users could configure that manually using registryUrls and hostRules like they already do for Maven.

Next, we should detect alternate registries within the gradle files without requiring registryUrls config, e.g.

buildscript {
    repositories {
        mavenCentral()
        mavenLocal()
        maven {
            url 'https://artifactory.private/artifactory/libs'
            credentials {
                username artifactoryUsername
                password artifactoryPassword
            }
        }
    }
}

In the above, we would registry the artifactory URL but ignore the credentials. The user would need to configure those via hostRules.

Finally, we should see if there are cases where we should parse/understand the credentials, e.g. if they are actually embedded within the project:

buildscript {
  repositories {
    maven {
           credentials {  username "xyz@x.com" password "abc" }
           url 'corpMvnURL'
    } 
  }
}

[rarkins]

Re: implementation what I'd like to do is a two-step process:

1. Parse build/settings files from raw to JS/JSON structure, i.e. in a JSON.parse()-like approach. But: we only parse objects/fields we care about
2. Apply logic to how we correlate those parsed files and build our array of dependencies

By doing this in two clear steps, we can (a) parse groovy and kotlin formats separately but share the same logic after that, and (b) swap in a "real" parser for the first step later if/when we need

[rarkins]

Time to revive this

[rarkins]

Example files:

• https://github.com/Jasig/uPortal/blob/master/build.gradle
• https://github.com/Jasig/NotificationPortlet/blob/master/build.gradle
• https://github.com/bagage/gradle-template/blob/master/build.gradle
• https://github.com/bagage/gradle-template/blob/master/module1/build.gradle
• https://github.com/apereo/cas/blob/master/build.gradle
• https://github.com/apache/fineract/blob/develop/fineract-provider/build.gradle

[rarkins]

Some questions I still have:

• When there are nested build.gradle files, can we simply "assume" that they are all related and belong to the parent project, or should we check for explicit links? If explicit, is it just include statements in settings.gradle, or elsewhere?

• Apart from build.gradle files, where else could dependency versions be defined? Is it restricted to gradle.properties, or can the filename be completely flexible? If flexible then in what places can it be specified?

• Are variables passed from parent down to child? e.g. everything defined in the parent's gradle.properties may be referenced in a child's build.gradle?

[jnizet]

When there are nested build.gradle files, can we simply "assume" that they are all related and belong to the parent project, or should we check for explicit links? If explicit, is it just include statements in settings.gradle, or elsewhere?

It would be bad practice to have an independant gradle project inside another independant gradle project, but nothing forbids it. A sub-project is indeed linked to its parent via the settings.gradle (or settings.gradle.kts) file.

Apart from build.gradle files, where else could dependency versions be defined? Is it restricted to gradle.properties, or can the filename be completely flexible? If flexible then in what places can it be specified?

It can be specified in many places:

• in build.gradle.kts files, if the Kotlin DSL is used rather than the groovy DSL
• in binary plugins, applied by the project
• in plugins applied by plugins (recursively)
• in other gradle files, applied by the project
• in build files which aren't named build.gradle
• in pom files

In short, a gradle build is not, at all, like a NodeJS package.json file. It's a program containing executable instructions, which can use plugins downloaded as Java libraries, and which, in the end, define, among many other things, dependencies of the project.

Most of the time, BTW, versions of dependencies aren't even specified directly in the dependencies. They are defined as variables in the program, or in libraries used by the program, etc.

Without actually using the gradle tooling API, you won't be able to have a correct set of dependencies and their version. It's way too dynamic.

[viceice]

I think variables or dependencies defined outside of the repo files can be ignored, as we can't update them anyways. So we only need to focus on variables and dependencies which are defined anywhere in the repo files.

[rarkins]

Thanks for the detailed response!

Without actually using the gradle tooling API, you won't be able to have a correct set of dependencies and their version. It's way too dynamic.

Unfortunately the gradle tooling API has proven itself to be unfit for purpose (slow, memory-hogging, unnecessarily downloading too much, and not giving us the precision we need), so we will be discontinuing using it and replacing with a simpler parser.

Remember: our goal is not to get the full list of dependencies and versions - it's just to get the list of dependency versions we can update.

I understand people can spray dependency definitions anywhere they wish, but that doesn't mean they should, and definitely doesn't mean we'll support it out of the box with Renovate. For most common use cases it can work out of the box, and for uncommon use cases people can use our regex manager to capture versions from whatever files they wish.

BTW when I said "where else could dependency versions be defined" I was meaning strings containing versions which are checked into the same repo. If dependency versions are from external plugins or plugins of plugins, we can't update those anyway - so they are not within scope.

Naturally we'll aim to support Kotlin the same as Groovy.

I would consider plugins out of scope (unless they're defined in the repo with their own build.gradle, in which case they're in scope already).

For other gradle files, is there really any good reason to spread dependency definitions across random files? Based on language-agnostic best software practices shouldn't they belong in centralized files like build.gradle or gradle.properties and not gradle/lets/me/put/things/anywhere/so/that_s/what/i_ll/do.gradle?

In summary, I know it's an interpreted language and people can put stuff in whatever location they want, but I'm not interested in supporting it, just like I'm not interested in supporting it if people base64 encode their dependency versions just because they can. Being able to scan a repository in seconds and not minutes is worth it, as is being able to automate dependency scanning in the first place. And people who insist on dependency versions in random places can still use our regex manager now afterall.

Anything I'm missing or being too stubborn on myself?

[jnizet]

Fine. Sorry if my response seemed harsh. You're right of course that you shouldn't care about versions you can't update.

For other gradle files, is there really any good reason to spread dependency definitions across random files?

The gradle project itself, for example, chooses to name its own gradle build files against the subproject name, to avoid having too many files named build.gradle[.kts]. See all the sub-projects in https://github.com/gradle/gradle/tree/master/subprojects. Many projects choose to split the main build.gradle file into separate files, each responsible for various parts of the build (documentation, publishing, whatever), using apply. This is common to avoid too large build files, and to share code between sub-projects.

Another extremely common thing to do, even in very simple build files, is to just use variables. Take for example this build file of a project of mine: https://github.com/Ninja-Squad/globe42/blob/master/backend/build.gradle.kts, using the most popular Java framework: Spring

As you can see,

• 4 plugins declared in the plugins block use the same kotlin version, defined in a variable on top.
• Most dependencies versions are automatically specified by the Spring Boot plugin
• One dependency uses a custom version thanks to the line ext["okhttp3.version"] = "4.6.0"
• The io.jsonwebtoken dependencies use the version specified in the variable jwtVersion defined before

Another very common way to share dependency versions between sub-projects is to define them as constants in a class in buildSrc.

So, of course, it would be better than nothing to try to parse the simplest common ways of defining dependencies, but you will miss many dependencies, even for simple projects.

[rarkins]

Thank you for these great examples. No problem at all with variables - I consider that to be a requirement from the start. Are variables only shared "down" from highest level in the repo to lowest, or can they also be shared up or sideways between components?

[renovate-release]

:tada: This issue has been resolved in version 23.97.0 :tada:

The release is available on:

• GitHub release
• 23.97.0

Your semantic-release bot :package::rocket: