How to add additional cache files?

[mumrah]

I'm trying out this action for Apache Kafka. We have a gradle task that generates Java source files. The generated files end up in some directory like src/main/generated. When building locally, Gradle will cache things as expected. If the input files don't change, it will skip that build step.

When using this build action, it always runs the "generate" Gradle task, since the output files don't exist. This causes downstream tasks to always run and never be UP-TO-DATE.

I tried caching the generated files explicitly with actions/cache@v3, but Gradle complains that it doesn't know where the output files came from:

2023-06-08T19:40:25.0584875Z 2023-06-08T19:40:24.142+0000 [INFO] [org.gradle.internal.execution.steps.ResolveCachingStateStep] Caching disabled for task ':metadata:processMessages' because:
2023-06-08T19:40:25.0586309Z   Gradle does not know how file 'src/generated/java/org/apache/kafka/common/metadata' was created (output property '$1'). Task output caching requires exclusive access to output paths to guarantee correctness (i.e. multiple tasks are not allowed to produce output in the same location).

How can I get this action to cache generated files in order to take advantage of incremental builds?

[Phlegethon90]

Don't put generated files in src/main/generated. Use a subdirectory in build and configure dependencies and/or tasks. That would be (semantically) correct. Does your task declare annotated outputs?

[mumrah]

I'm not sure exactly what you mean by annotated outputs, but we do have inputs and outputs declared in these tasks. Here is what one looks like:

task processMessages(type:JavaExec) {
    mainClass = "org.apache.kafka.message.MessageGenerator"
    classpath = configurations.generator
    args = [ "-p", "org.apache.kafka.common.metadata",
             "-o", "src/generated/java/org/apache/kafka/common/metadata",
             "-i", "src/main/resources/common/metadata",
             "-m", "MessageDataGenerator", "JsonConverterGenerator",
             "-t", "MetadataRecordTypeGenerator", "MetadataJsonConvertersGenerator"
           ]
    inputs.dir("src/main/resources/common/metadata")
        .withPropertyName("messages")
        .withPathSensitivity(PathSensitivity.RELATIVE)
    outputs.cacheIf { true }
    outputs.dir("src/generated/java/org/apache/kafka/common/metadata")
  }

Why would moving the generated sources to build change anything? Is "build" special in some way? From looking at caching debug from this action, it is only caching things in Gradle Home, not any of the project directories.

Here's one of our module configurations https://github.com/apache/kafka/blob/trunk/build.gradle#L1168-L1293

Do you know of a project using this action that has generated sources (grpc/avro/etc?) Maybe I could figure this out with a working example.

[mumrah]

Ok, after moving the generated sources into a "build" directory, it does seems to get the task result from the cache

> Task :metadata:processMessages FROM-CACHE
Watching 61 directories to track changes
Build cache key for task ':metadata:processMessages' is 592f897b9dbd2b252c37b2b165079dbe
Task ':metadata:processMessages' is not up-to-date because:
  No history is available.
Watching 61 directories to track changes
Watching 61 directories to track changes
Loaded cache entry for task ':metadata:processMessages' with cache key 592f897b9dbd2b252c37b2b165079dbe

Is this "Task ':metadata:processMessages' is not up-to-date because: No history is available." expected in this case?

I would still like to understand the "build" thing. Is this directory special to Gradle?

Anyways, I think it's working. Thanks!

[Phlegethon90]

I'm not sure exactly what you mean by annotated outputs, but we do have inputs and outputs declared in these tasks.

I thought, that you implemented a dedicated task for this (by using ExecOpertations.javaexec in an extended DefaultTask or overriding property-getters in a custom task extending JavaExec; see here). But registering a JavaExec-task and declaring outputs like this is also ok.

Why would moving the generated sources to build change anything? Is "build" special in some way?

The directory build is the default value of the property layout.buildDirectory, so special in (at least) that way. I think, that caching of (subdirectories of) Gradle user home is somehow related to build.

Do you know of a project using this action that has generated sources (grpc/avro/etc?) Maybe I could figure this out with a working example.

Not a public repo, sorry.

Is this "Task ':metadata:processMessages' is not up-to-date because: No history is available." expected in this case?

I think, that the word "up-to-date" is meant to be understood in the context of incremental build and task outcomes. So no history in "up-to-date" doesn't mean no "from-cache".

[bigdaz]

When you use the gradle-build-action, you do a fresh checkout of your source repository and then run Gradle on it. Because of this, you generally won't see any tasks UP-TO-DATE: you only see this status when you re-run a build in the same location, and the outputs are still in place from a previous build.

You can see FROM-CACHE tasks, since the local build cache is stored in the Gradle User Home directory, and this is saved/restored by the gradle-build-action.

To get a better understanding of how this all works, check out the Incremental Build chapter of the Gradle user guide.