Open TimvdLippe opened 5 months ago
There's a viewer for the configuration cache data: https://github.com/gradle/gcc2speedscope, which may give some insights into what is being stored. @TimvdLippe could you please try that?
I will need security approval to run such tooling, so it will take me a couple of days to do so, but will do 👍
Is there a workaround to disable the configuration cache for the task anyways? I think we can mark it as incompatible ourselves, as we are an outlier in terms of codebase size and then we can wait for a potential fix.
Is there a workaround to disable the configuration cache for the task anyways?
There's Task.notCompatibleWithConfigurationCache, you can look how Gradle itself uses it for a similar purpose. This should be enough to at least prevent loading the cache. I'm not sure if it also disables storing the graph (Gradle still checks for CC errors coming from other tasks), but that's the best option I know of.
I was able to clone the project and build it. Unfortunately I am running into runtime issues with a Suppressed: kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
and later a Caused by: java.lang.ClassNotFoundException: gcc2speedscope.EventStore$queryProfiles$1
. I lack Kotlin knowledge and am not sure if these are pre-existing issues or related to our internal infrastructure. Will try to debug further with a colleague who has knowledge on Kotlin
All right. The failures were related to some internal changes I had to make. Fortunately I was able to run with the assistance of a colleague.
Some data:
debug.log
was 16GBspeedscope.json
is 1.1MBWith an eyeball look (since we have loads and loads of modules), I was able to see that some tasks stood out:
graphViewMain
which is 115Kb for a relatively small modulegraphViewTest
which is 175KB for that same modulegraphView
for each of our sourcesetsgraphViewX
takes 1MB and we have about 5 of these, so 5MB per moduleOther tasks are also large, but at least from a first glance graphViewX
appears to be the culprit. When I omit the data for all these, we are still looking at a large number, but maybe the findings from this task can help us with the rest as well.
Looking at the PR in question, I wonder why we annotated compileFiles
and runtimeFiles
as @InputFiles
? In our internal plugins, we have some tasks that also operate on our classpaths, but they don't have such a large configuration input. Instead, they operate on a @Classpath
and we retrieve the files during task execution. Since we already have the runtimeClassPath
property, shouldn't we annotate that property with @Classpath
and then retrieve the files?
Example from our internal plugin that afaik doesn't have the same configuration cache size hit:
@Classpath
public abstract ConfigurableFileCollection getSourcesClasspath();
@mlopatkin Is that sufficient information for you or do you want me to run anything else?
To unblock our upgrade, we will mark the graphView
tasks as incompatible with the configuration cache. Hopefully in the future the situation improves and we can use the configuration cache again for these tasks.
This is the workaround I used if anybody else runs into this issue:
import com.autonomousapps.tasks.GraphViewTask
project.getPluginManager().withPlugin('com.autonomousapps.dependency-analysis', { plugin ->
project.getTasks().withType(GraphViewTask).configureEach {
it.notCompatibleWithConfigurationCache("https://github.com/autonomousapps/dependency-analysis-gradle-plugin/issues/1098")
}
}
Build scan link Internal scan unfortunately
Plugin version Latest master (1.28.0)
Gradle version 8.6
JDK version 17
Describe the bug When trying to integrate the latest version of the dependency analysis plugin into our codebase, we observed super slow configuration phases with Gradle. I ran
./gradlew projectHealth
on my laptop and this is the result:Before, the largest configuration cache side I had on my machine was 24MB. Gradle spent about 2 minutes in the "Storing configuration cache state" and then proceeded to run for 20 minutes in "Loading configuration cache state" after which I killed the process as I didn't expect it to finish.
Unfortunately the caches themselves are
.bin
files, so I can't give you anymore information of which particular part of the cache is large. I also suspect the size of the configuration cache correlates with the number of Java files, as we configure them in file collections.To Reproduce Steps to reproduce the behavior:
./gradlew projectHealth
Expected behavior The configuration cache for a run that includes
projectHealth
is within reasonable bounds compared to other tasks. In our case, we expect the cache to be smaller than 30 MB. It also should not cause the configuration cache storing and loading to take minutes for large builds. For a smaller build of ours, it still took 6 seconds to load the full configuration cache, where the entry is 211MB (about 1/4th of all modules included in this build, but typically excludes the largest ones we have).CC @cobexer and @ljacomet who contributed the configuration cache fixes in #1039