Question: How to handle published interdependent libraries

[zfreeds]

Hi, I love the videos and they've been very helpful for understanding Gradle. One issue I've been thinking about a lot is: What's the best way to handle the versioning of interdependent (private) libraries owned by different teams? For example:

Upgrading Transitive Dependencies Leads to NoSuchMethod

I'm noticing issues like NoSuchMethod being thrown if a consumer wants to use an upgraded version of a dependency other libraries depend on. I'm thinking that strict version constraints to MAJOR versions can help fail earlier but that only works as long as no one accidentally releases a minor/patch version that's not backwards-compatible. Honestly, I'm so surprised this isn't a problem affecting public dependencies.

I imagine you will suggest Version Catalogs or Platforms - does that require one team own/verify it? Is there a way to test that all the libraries are compatible with eachother and/or backwards-compatible with themselves?

Supporting Multiple Variants

Some libraries still need to support Spring Boot 2 vs Spring Boot 3. Would you suggest using feature variants? This might require two separate platforms or catalogs. Currently, we have core libraries and a library for each variant like micronaut/spring.

[jjohannes]

Hi @zfreeds. Thank you for the feedback. Here are my thoughts on the two topics:

Upgrading Transitive Dependencies

There are multiple tools for this. For example, Gradle itself uses a customized solution based on japicmp to ensure no public API is changed during minor releases. There exists a Gradle plugin that is used there: https://github.com/melix/japicmp-gradle-plugin There are also other alternative tools but I haven't used any myself.

Don't know if this helpful, but there is 2017 talk from Netflix who deal with this large scale: https://www.youtube.com/watch?v=k_mPS_1JpXM

Supporting Multiple Variants

The topic sounds like the most elegant solution would be variants with attributes. The wording is confusing. Feature Variants (as documented here) are distinguished by capability and you explicitly select, for each Module independently, which variant you want/need. With attributes, you can tell Gradle to select a certain variant for all modules in the dependency graph. Similar to how Gradle selects different variants for compile time and runtime.

Their is no convenience for this built-in right now, but you can use the registerFeature with a few adjustments on the producer (library) side. Something like:

val springBoot2 = sourceSets.create("springBoot2")
java.registerFeature(springBoot2.name) {
    usingSourceSet(springBoot2)
}

val springBootVersion = Attribute.of("org.example.spring.boot-version", Int::class.javaObjectType)

configurations.apiElements {
    attributes.attribute(springBootVersion, 3)
}
configurations.runtimeElements {
    attributes.attribute(springBootVersion, 3)
}
configurations.getByName(springBoot2.apiElementsConfigurationName) {
    // As we want selection purely based on attributes, add back the "default" capability
    outgoing.capability(provider { "${project.group}:${project.name}:${project.version}" }) 
    attributes.attribute(springBootVersion, 2)
}
configurations.getByName(springBoot2.runtimeElementsConfigurationName) {
    outgoing.capability(provider { "${project.group}:${project.name}:${project.version}" })
    attributes.attribute(springBootVersion, 2)
}

Users of the library then have the possibility to select a variant for the whole dependency tree. For modules that do not have the different variants, the attribute is ignored.

val springBootVersion = Attribute.of("org.example.spring.boot-version", Int::class.javaObjectType)
sourceSets.all { 
    configurations.getByName(compileClasspathConfigurationName) {
        attributes.attribute(springBootVersion, 2)
    }
    configurations.getByName(runtimeClasspathConfigurationName) {
        attributes.attribute(springBootVersion, 2)
    }
}

[zfreeds]

Hey @jjohannes, that was extremely helpful! Looking into japicmp lead me to the official binary-compatibility-validator which looks like it can be easily added to any kotlin library. I was worried since some of those links involve whole implementations/systems to support (e.g the netflix video). This is exactly what I was looking for. Thank you for also clarifying on variants with attributes vs feature variants.

I might have missed it, but I don't see your opinion on whether libraries should add constraints based on library major versions or not and I'm very curious to hear your thoughts here.

[jjohannes]

@zfreeds happy to hear that this helped you.

I might have missed it, but I don't see your opinion on whether libraries should add constraints based on library major versions or not and I'm very curious to hear your thoughts here.

I missed that. Yes I think it is useful to do that if you know that the library you depend on follows semantic versioning and a update will very likely lead to a broken overall setup. Without that Gradle just updates to new major versions if there is a conflict (as it does not know about semantic versioning).

There are different ways to express such things with Gradle's rich versions. I would probably do something like this:

dependencies {
  implementation("com.sun.mail:jakarta.mail:1.6.7") {
    version { reject("[2.0.0,)") } // constraint to cause a conflict between 1.x and 2.x to fail because you forbid 2+ versions
  }
}

[jjohannes]

For avoiding API breakages, this Gradle plugin could also be interesting: https://github.com/palantir/gradle-revapi