Project subtree configuration with Isolated Projects

[alllex]

Gradle currently provides APIs like project.subprojects and project.allprojects to apply configuration to a part of a project tree. These APIs expose mutable state of other projects to the parent project that calls those methods. The callbacks also share any state captured in the closures that can be mutable.

This makes build logic based on those APIs prone to become incompatible with Isolated Projects.

For some time now, Gradle provided a recommended way to share build logic across projects via convention plugins and discouraged cross-project configuration. Since convention plugins are applied explicitly and directly to a project, by themselves they do not cause isolation problems.

Since Gradle 8.8, there is also gradle.lifecycle.beforeProject {} IP-compatible API that could be used for subtree configuration. It's primary use is for cases when build logic needs to be applied to all projects. Additional filtering inside the callbacks can technically simulate subtree configurations.

At this point, we are interested in collecting use cases and requirements for Isolated Projects-safe subtree configuration.

[martinbonnin]

We have a use case in Apollo that requires establishing a bidirectional dependency between 2 modules (Apollo doc, Gradle issue). I haven't found a way yet to make this project isolation compatible without having the user write the bidirectional dependency manually or relying on lenient configurations.

Other use cases that come to mind:

• Publishing (see nmcp)
• Dokka
• BCV

All that being said, I thought that Gradle 8.8 gradle.lifecycle.beforeProject {} (issue comment) would be a replacement for allProjects {} that would be project isolation compatible? Should we avoid that? Any guidance how to handle project isolation is warmly welcome.

[alllex]

I thought that Gradle 8.8 gradle.lifecycle.beforeProject {} (https://github.com/gradle/gradle/issues/22514#issuecomment-2099179052) would be a replacement for allProjects {} that would be project isolation compatible? Should we avoid that?

It is mainly a replacement for allprojects {} applied in the root build, because the callback would run for every project unconditionally. The current issue is targeted at builds that might find the need to insert subtree configuration in non-root projects.

An example of a project tree that illustrates that:

root
\ microservices <-- uses 'subprojects' to configure all microservices
  \ micro1
  \ micro-group2
   \ micro2-1
\ libraries

Technically, you could add filtering and repurpose the new lifecycle callbacks for subtree configuration. Though, it means that all the logic would have to live in the settings script, which might or might not be desirable.

Thank you for calling this out, I added a paragraph on this to the issue description.

[martinbonnin]

Gotcha. Thanks for the update 👍

[mathjeff]

Would this issue include making it more convenient for one task to depend on the aggregate of all artifacts of a certain type declared in any project?

I put more details into https://github.com/gradle/gradle/issues/29403

[autonomousapps]

This blog post was inspired by a real use-case in Cashapp's backend services. The post describes doing it from the root, but in the real use-case, we have different configurations for different subtrees. We manage this in an "IP-safe" way via a build service, but as you can see the code to get it to work safely is complex and relies on some assumptions, and also relies on build services working correctly, which they don't always.

At Cash, we're experimenting with migrating many repos into a monorepo. Each individual repo has individual configuration that is challenging to reconcile with others in the monorepo context. We've built plugins to allow some differentiation between the various subtrees.

At Square in the Android monorepo, we ran into many issues with subprojects blocks that were configuring sub-trees for various reasons. We also had to build complex plugins to handle that in a "safe" way. It would indeed have been much easier, and saved hundreds (easily) of build-engineer-hours, to have a safe subprojects block instead.

One example of subtree-specific configuration in the Android monorepo was that certain subtrees were disallowed from depending on other subtrees. The ability to forbid dependency edges in a monorepo is critical for their utility.

I have a question. With the new before/afterproject blocks, what's the classloader being used by code in those blocks? The one associated with the settings script? The one associated with the subproject build script? This issue with classloaders is one of the reasons I avoid sub/allproject blocks like the plague.

[joshfriend]

AGP applies some version compatibility checks using allprojects { } (example AGP issue and related gradle issue) that could possibly be migrated to the new API

[lwasyl]

One use case for configuring subprojects is to have a convention plugin that sets ups multiple child projects using other convention plugins. For example consider a project where each feature has a well-established set of modules:

 - feature-foo
   - api
   - domain
   - presentation
   - test-fixtures
   - tests
   - ...
 - feature-bar
   - api
   - domain
   - presentation
   - test-fixtures
   - tests
   - ...

each module with the same name applies the same convention plugin to set up this type of project (applying either kotlin or android plugin, adding common dependencies, applying other appropriate convention plugins etc.). Ideally, feature-* module would itself have a feature plugin that would automatically apply correct plugins to all the subprojects. Would that be a use case for Isolated Projects-safe subtree configuration?

[bingranl]

Android Studio register some models by applying a plugin with allProject block in the generated init script. I think the reason why we want to apply those models from android studio is to make them depend on AS versions instead of AGP version. There might be other reason though.

With that, from AS point of view, if there is an tooling api to register additional models for all projects, we would be quite happy about that.

context https://issuetracker.google.com/issues/253076415

[bamboo]

Android Studio register some models by applying a plugin with allProject block in the generated init script.

@bingranl that could be done via a Settings / Gradle plugin (or an init script as well) using GradleLifecycle#beforeProject.

[alllex]

I have a question. With the new before/afterproject blocks, what's the classloader being used by code in those blocks? The one associated with the settings script? The one associated with the subproject build script? This issue with classloaders is one of the reasons I avoid sub/allproject blocks like the plague.

@autonomousapps, the isolated lifecycle callbacks use the classloader of the script they are registered in. When they are registered in the settings script, then it's the setting script's classloader. In can also be an init script. This also makes it possible to have a type that is "private" to the script, and yet it can be shared between the beforeProject and afterProject callbacks.

[alllex]

Would this issue include making it more convenient for one task to depend on the aggregate of all artifacts of a certain type declared in any project?

@mathjeff, not necessarily. The use case you provide will be address as part of https://github.com/gradle/gradle/issues/25179