Support Gradle configuration cache on multiple JVMs

[nedtwigg]

Gradle configuration cache requires task state to be round-trip serializable. As outlined at #987, this is a signficant constraint on plugin developers and it slows down the end-user experience of change-test-change-test on a single JVM daemon. We built a workaround (JvmLocalCache) that complies with Gradle's requirement by making our tasks round-trip serializable without requiring much work from plugin developers or the end user's machine, but it only works within a single JVM.

To work on multiple JVMs, we need to

• make every step round-trip serializable
• remove JvmLocalCache

[eskatos]

FYI Gradle 8 will do a full roundtrip on the first invocation with CC enabled. See work in progress here https://github.com/gradle/gradle/issues/21985#issuecomment-1260018610 The goal is for this to be the default in 8.0.

[nedtwigg]

Full roundtrip will work great on the same JVM, so Spotless will still work and not work under the same conditions it does currently. EDIT: Compatibility with configuration cache within a single JVM on Gradle 8 is confirmed.

[nedtwigg]

I think we now have a way to incrementally build this until we can eventually remove our workaround. See:

• https://github.com/diffplug/spotless/pull/1945

In this comment, I'm going to maintain a list of remaining TODOs before we will fully support Gradle's configuration cache model without any workarounds. If you'd like to help: comment in this issue when you start on a given step, follow-up with a PR (draft is fine) and I will update the checklist below

• [x] remove LineEnding.GIT_ATTRIBUTES (at least for Gradle) because very difficult to make it serializable #1274
  • we can keep GIT_ATTRIBUTES_FAST_ALLSAME which is the default anyway
• [x] lib(generic.PipeStepPair) #1954
• [x] lib(generic.EndWithNewlineStep)
• [x] lib(generic.IndentStep) #1945
• [x] lib(generic.Jsr223Step) #2039
• [x] lib(generic.LicenseHeaderStep) #2132
• [x] lib(generic.NativeCmdStep) #2132
• [x] lib(generic.ReplaceRegexStep)
• [x] lib(generic.ReplaceStep)
• [x] lib(generic.TrimTrailingWhitespaceStep)
• [x] lib(groovy.RemoveSemicolonsStep)
• [x] lib(antlr4.Antlr4FormatterStep) #2033
• [x] lib(cpp.ClangFormatStep) #1955
• [x] extra(cpp.EclipseFormatterStep) #1950
• [x] lib(gherkin.GherkinUtilsStep) #2039
• [x] extra(groovy.GrEclipseFormatterStep) #1950
• [x] lib(java.GoogleJavaFormatStep) #2033
• [x] lib(java.ImportOrderStep) #2033
• [x] lib(java.PalantirJavaFormatStep) #2033
• [x] lib(java.RemoveUnusedImportsStep) #2033
• [x] extra(java.EclipseJdtFormatterStep) #1950
• [x] lib(java.FormatAnnotationsStep) #2033
• [x] lib(java.CleanthatJavaStep) #2033
• [x] lib(json.gson.GsonStep) #2039
• [x] lib(json.JacksonJsonStep) #2039
• [x] lib(json.JsonSimpleStep) #2039
• [x] lib(json.JsonPatchStep) #2039
• [x] lib(kotlin.KtLintStep) #2033
• [x] lib(kotlin.KtfmtStep) #2033
• [x] lib(kotlin.DiktatStep) #1945
• [x] lib(markdown.FreshMarkStep) #2039
• [x] lib(markdown.FlexmarkStep) #2039
• [x] lib(npm.EslintFormatterStep) #2135
• [x] lib(npm.PrettierFormatterStep) #2135
• [x] lib(npm.TsFmtFormatterStep) #2135
• [x] lib(pom.SortPomStepStep) #2039
• [x] lib(protobuf.BufStep) #1955
• [x] lib(python.BlackStep) #1955
• [x] lib(biome.BiomeStep) #2135
• [x] lib(scala.ScalaFmtStep) #2033
• [x] lib(sql.DBeaverSQLFormatterStep) #2132
• [x] extra(wtp.EclipseWtpFormatterStep) #1952
• [x] lib(yaml.JacksonYamlStep) #2039

[lehmanju]

I checked npm.EslintFormatterStep, npm.PrettierFormatterStep, npm.TsFmtFormatterStep and sql.DBeaverSQLFormatterStep. They all use StepHarness for testing which has roundtrip serialization testing enabled by default. Since their tests do not fail, it should mean they are already serializable, am I correct?

For NativeCmdStep I created a PR #2108 which adds a test using StepHarness.

[nedtwigg]

Thanks mostly to @Goooler's tireless dedication, we have finally shipped:

• plugin-gradle 3.0.0.BETA1
• plugin-maven 2.44.0.BETA1

These contain a bazillion fixes. Please test and let us know if you encounter any issues!

[marcphilipp]

JUnit build updated ✅

[bddckr]

All working fine for me as well, except I used

removeUnusedImports("cleanthat-javaparser-unnecessaryimport")

as seen in the docs. That one seems to not support the config cache yet:

Could not load the value of field `steps` of task `:spotlessInternalRegisterDependencies` of type `com.diffplug.gradle.spotless.RegisterDependenciesTask`.
> Could not load the value of field `excluded` of `com.diffplug.spotless.java.CleanthatJavaStep` bean found in field `roundtripStateInternal` of `com.diffplug.spotless.FormatterStepSerializationRoundtrip` bean found in field `steps` of task `:spotlessInternalRegisterDependencies` of type `com.diffplug.gradle.spotless.RegisterDependenciesTask`.

I just removed it as it looks like googleJavaFormat already takes care of that.

[Magotchi]

I get a similar error to @bddckr if I use any prettier steps, and only while using the Gradle Configuration Cache, like:

Could not load the value of field `steps` of task `:spotlessJava` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.
> Could not load the value of field `additionalNpmrcLocations` of `com.diffplug.spotless.npm.NpmPathResolver` bean found in field `resolver` of `com.diffplug.spotless.npm.NpmFormatterStepLocations` bean found in field `locations` of `com.diffplug.spotless.npm.PrettierFormatterStep$State` bean found in field `roundtripStateInternal` of `com.diffplug.spotless.FormatterStepSerializationRoundtrip` bean found in field `steps` of task `:spotlessJava` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.

or:

Could not load the value of field `steps` of task `:spotlessPrettierGeneric` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.
> Could not load the value of field `additionalNpmrcLocations` of `com.diffplug.spotless.npm.NpmPathResolver` bean found in field `resolver` of `com.diffplug.spotless.npm.NpmFormatterStepLocations` bean found in field `locations` of `com.diffplug.spotless.npm.PrettierFormatterStep$State` bean found in field `roundtripStateInternal` of `com.diffplug.spotless.FormatterStepSerializationRoundtrip` bean found in field `steps` of task `:spotlessPrettierGeneric` of type `com.diffplug.gradle.spotless.SpotlessTaskImpl`.

I've worked around it by declaring Spotless incompatible with the configuration cache, like:

tasks.withType<SpotlessTask>().configureEach {
  notCompatibleWithConfigurationCache(
      "Spotless 7.0.0.BETA1 with prettier steps does not support the configuration cache.")
}

[davidburstrom]

FWIW, I've upgraded to 7.0.0.BETA2 in all my repositories and everything seems to be in order! Great job!