text resource from archive entry gets deleted on clean

[Vampire]

I have the following in my build script:

tasks.withType<SpotBugsTask> {
    reports {
        xml.isWithMessages = true
        html.let { it as CustomizableHtmlReport }.stylesheet = resources.text.fromArchiveEntry(spotbugsStylesheets, "fancy-hist.xsl")
    }

    finalizedBy(tasks.register("${name}HtmlReport") {
        val stylesheetFile = reports.html.let { it as CustomizableHtmlReport }.stylesheet!!.asFile()
        inputs.file(stylesheetFile).withPropertyName("spotbugsStylesheet").withPathSensitivity(NONE)
        val input = reports.xml.destination
        inputs.files(fileTree(input)).withPropertyName("input").withPathSensitivity(NONE).skipWhenEmpty()
        val output = file(input.absolutePath.replaceFirst(Regex("\\.xml$"), ".html"))
        outputs.file(output).withPropertyName("output")

        @Suppress("UnstableApiUsage")
        doLast("generate spotbugs html report") {
            TransformerFactory.newInstance("net.sf.saxon.TransformerFactoryImpl", TransformerFactoryImpl::class.java.classLoader)
                    .newTransformer(StreamSource(stylesheetFile))
                    .transform(StreamSource(input), StreamResult(output))
        }
    })
}

Now if I call ./gradlew clean check, I get

> Task :spotbugsMainHtmlReport FAILED

FAILURE: Build failed with an exception.

* What went wrong:
A problem was found with the configuration of task ':spotbugsMainHtmlReport'.
> File 'D:\Sourcecode\other\command-framework\build\tmp\expandedArchives\spotbugs-4.0.0-beta1.jar_f5afb19f10f56e94cbb465bf3dc8f909\fancy-hist.xsl' specified for property 'spotbugsStylesheet' does not exist.

So it seems during configuration the archive is extracted, during the clean it is deleted and then when it is needed it is not present anymore. If I do first a clean and then a check as separate runs it works fine.

[ethnhll]

I can confirm that this occurs for me as well for the usage that @Vampire is encountering.

I can also reproduce this issue by loading a TextResource via URI (https://myremotefile.com/resource.xml for example) and then converting that resource to a file via .asFile()

(Paths have been changed to protect the innocent)

checkstyle {
    toolVersion = '8.20'
    configFile = resources.text.fromUri("https://myremotefile.com/resource.xmll").asFile()
    check.setDependsOn(check.taskDependencies.getDependencies() - checkstyleMain - checkstyleTest)
}

Calling gradle clean check yields...

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':checkstyleMain'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.accept(ExecuteActionsTaskExecuter.java:151)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.accept(ExecuteActionsTaskExecuter.java:148)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:191)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:141)
        at org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionStateTaskExecuter.execute(ResolveBeforeExecutionStateTaskExecuter.java:75)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:108)
        at org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionOutputsTaskExecuter.execute(ResolveBeforeExecutionOutputsTaskExecuter.java:67)
        at org.gradle.api.internal.tasks.execution.ResolveAfterPreviousExecutionStateTaskExecuter.execute(ResolveAfterPreviousExecutionStateTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:94)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:95)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:73)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:49)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:49)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:43)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:355)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:343)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:336)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:322)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:134)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker$1.execute(DefaultPlanExecutor.java:129)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:202)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:193)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:129)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
....
Caused by: com.puppycrawl.tools.checkstyle.api.CheckstyleException: Unable to find: /home/ethnhl/test-project/build/tmp/resource/wrappedInternalText5323175846595342341.txt

[ethnhll]

@Vampire Looking around, I found a reference to an issue similar to this on the Gradle forums. https://discuss.gradle.org/t/extract-resources-from-plugin-jar/30949

Seems that we are trying to resolve a File during the configuration phase of the task, which is then cleaned during the execution phase of clean. When running gradle clean build, configuration only happens once (before clean) and does not happen again, so the File resource defined in configuring the task cannot be found during build's execution. This is why if you run the two separately, gradle clean then gradle check, it works because gradle check reconfigures your task which defines the file resource.

Execution occurs during doLast{...} so the line val stylesheetFile = reports.html.let { it as CustomizableHtmlReport }.stylesheet!!.asFile() occurring outside of that closure will not work following clean.

I would simply reference the stylesheet as a TextResource and then within doLast{...} resolve the resource as a File.

[Vampire]

Ah, good idea, it resolves it twice then though, one time during configuration for the inputs.file and one time during execution, but I guess any solution would probably yield the same behavior.

[Vampire]

Oh, no, it does not work and cannot work. It is not the doLast where it fails. The complaint is about the input file for the task in the line inputs.file(stylesheetFile).withPropertyName("spotbugsStylesheet").withPathSensitivity(NONE).

Even if I use the following:

tasks.withType<SpotBugsTask> {
    reports {
        xml.isWithMessages = true
        html.let { it as CustomizableHtmlReport }.stylesheet = resources.text.fromArchiveEntry(spotbugsStylesheets, "fancy-hist.xsl")
    }

    finalizedBy(tasks.register("${name}HtmlReport") {
        val stylesheet = reports.html.let { it as CustomizableHtmlReport }.stylesheet!!
        inputs.file(stylesheetFile.asFile()).withPropertyName("spotbugsStylesheet").withPathSensitivity(NONE)
        val input = reports.xml.destination
        inputs.files(fileTree(input)).withPropertyName("input").withPathSensitivity(NONE).skipWhenEmpty()
        val output = file(input.absolutePath.replaceFirst(Regex("\\.xml$"), ".html"))
        outputs.file(output).withPropertyName("output")

        @Suppress("UnstableApiUsage")
        doLast("generate spotbugs html report") {
            TransformerFactory.newInstance("net.sf.saxon.TransformerFactoryImpl", TransformerFactoryImpl::class.java.classLoader)
                    .newTransformer(StreamSource(stylesheetFile.asFile()))
                    .transform(StreamSource(input), StreamResult(output))
        }
    })
}

I get the same error as before.

[Vampire]

This is the proper work-around:

tasks.withType<SpotBugsTask> {
    reports {
        xml.isWithMessages = true
        html.let { it as CustomizableHtmlReport }.stylesheet = resources.text.fromArchiveEntry(spotbugsStylesheets, "fancy-hist.xsl")
    }

    finalizedBy(tasks.register("${name}HtmlReport") {
        val stylesheet = reports.html.let { it as CustomizableHtmlReport }.stylesheet!!
        inputs.property("spotbugsStylesheet", stylesheet.asString())
        val input = reports.xml.destination
        inputs.files(fileTree(input)).withPropertyName("input").withPathSensitivity(NONE).skipWhenEmpty()
        val output = file(input.absolutePath.replaceFirst(Regex("\\.xml$"), ".html"))
        outputs.file(output).withPropertyName("output")

        @Suppress("UnstableApiUsage")
        doLast("generate spotbugs html report") {
            TransformerFactory.newInstance("net.sf.saxon.TransformerFactoryImpl", TransformerFactoryImpl::class.java.classLoader)
                    .newTransformer(StreamSource(stylesheetFile.asFile()))
                    .transform(StreamSource(input), StreamResult(output))
        }
    })
}

[ethnhll]

You shouldn't need to specify the property on inputs there.

I have the following which works for me (though I am only customizing SpotbugsMain).

import javax.xml.transform.TransformerFactory
import javax.xml.transform.stream.StreamResult
import javax.xml.transform.stream.StreamSource

import static org.gradle.api.tasks.PathSensitivity.NONE

configurations {
    spotbugsStylesheets {
        transitive = false
    }
}

dependencies {
    spotbugsPlugins 'com.h3xstream.findsecbugs:findsecbugs-plugin:1.7.1'
    spotbugsPlugins 'com.mebigfatguy.fb-contrib:fb-contrib:7.4.3.sb'
    spotbugsStylesheets "com.github.spotbugs:spotbugs:$spotbugs.toolVersion"
}

spotbugs {
    toolVersion = '3.1.12'
    effort = 'max'
    spotbugsMain.reports {
        xml.withMessages = true
        html.stylesheet = resources.text.fromArchiveEntry(configurations.spotbugsStylesheets, 'fancy-hist.xsl')
    }
}

spotbugsMain {
    task "SpotBugsHtmlReport" {
        def input = reports.xml.destination
        inputs.files fileTree(input) withPropertyName 'input' withPathSensitivity NONE skipWhenEmpty()
        def output = file(input.absolutePath.replaceFirst(/\.xml$/, '.html'))
        outputs.file output withPropertyName 'output'
        doLast {
            def factory = TransformerFactory.newInstance()
            def transformer = factory.newTransformer(new StreamSource(reports.html.stylesheet.asFile()));
            transformer.transform(new StreamSource(input), new StreamResult(output))
        }
    }
    spotbugsMain.finalizedBy "SpotBugsHtmlReport"
}

[Vampire]

But how should this correctly work? The stylesheet needs to be input. Or the task will be up-to-date even if the stylesheet changed.

[ethnhll]

You are referencing the stylesheet from the spotbugs jar, so the only way it would change is if the spotbugs version is changed.

[Vampire]

So? That wouldn't trigger the task to be not up-to-date, would it? Except the new version would also generate a different XML report. Assuming the XML report is the same, the task will be up to date, even if the stylesheet changed. And besides that, due to updates in the stylesheet I currently even do not use the same Spotbugs version for analysis and stylesheets, but a newer beta version for the latter.

[mateuszkwiecinski]

I'm experiencing exactly the same issue as the original one. The suggested workaround is to declare the configuration as a task input, but that doesn't seem to be a proper solution if the resource file needs to be used to configure an extension. I searched through Gradle documentation but haven't found anything saying that would be an intended behavior, is it?

In my case I'm using a Gradle plugin that configures library based on a config file added as project resource. Content of published jar: Published plugin loads the detekt-config.yml file and configures extension registered by Detekt plugin used by detekt task when my own plugin is being applied:

val config: ConfigurableFileCollection = extension.config
config.setFrom(resources.text.fromArchiveEntry(jarPath, "detekt-config.yml"))

See the full code here.

Calling regular ./gradlew clean + ./gradlew detekt works as expected, but caling ./gradlew clean detekt makes the build fail as the tmp/expandedArchives gets removed (together with whole build). It is enough to call standalone ./gradlew help to re-create the expandedArchives folder with proper config.

Is that something that can be fixed by Gradle or simply we shouldn't read resource file from archive in configuration phase (as it happens only once for single gradle execution)?

[henrpe]

I propose a workaround solution for the ./gradlew clean build problem for checkstyle with custom rules inside a Java library (jar file). The jar file contains a directory called checkstyle which contains checkstyle.xml and header-java-regex.checkstyle files. I created a new Copy task for extracting this configuration called unzipCheckstyleConfig which is run before all Checkstyle type tasks. This completely bypasses the problem when using resources.text.fromArchiveEntry(..) as checkstyle configuration is extracted after clean has run. I would imagine the following solution can be used in other cases as well as a replacement for resources.text.fromArchiveEntry(..). Tested with Gradle 6.4.1.

As our checkstyle rules also validate file header license then these settings are also in the following example. If you don't need such validations then you can ignore the parts regarding currentYear, headerFile and configProperties.

plugins {
    id 'checkstyle'
    ..   
}

configurations {
    checkstyleConfig
    ..
}

dependencies {
    checkstyleConfig 'groupId:artifactId:1.0.0'
    ..  
}

checkstyle {
    toolVersion '8.32'

    def customConfigFile = file("$buildDir/tmp/checkstyle/checkstyle.xml")
    def currentYear = java.time.Year.now()
    def headerFile = file("$buildDir/tmp/checkstyle/header-java-regex.checkstyle")

    configFile = customConfigFile
    configProperties = [
            'current.year'          : currentYear,
            'checkstyle.header.file': headerFile
    ]
}

tasks.withType(Checkstyle) {
    dependsOn 'unzipCheckstyleConfig'
    inputs.dir("$buildDir/tmp/checkstyle")
}

task unzipCheckstyleConfig(type: Copy) {
    from zipTree(configurations.checkstyleConfig.singleFile)
    into "$buildDir/tmp"
    include 'checkstyle/**'
}

[Vampire]

I was just hit by this again. :-( This time it was for a property of a task that is a ConfigurableFileCollection.

When I had

configurableFileCollection.from(
    resources.text.fromUri(getClass().getResource('...')).asFile('UTF-8'),
    file('...')
)

the same problem was present when using with clean so that the file was missing.

This is not a viable solution as then each time the file collection is queried a new file is generated which disturbs the tactic of incremental processing of that task.

configurableFileCollection.from(
    { resources.text.fromUri(getClass().getResource('...')).asFile('UTF-8') },
    file('...')
)

I basically had to do the same work-around conceptually:

task unpackFile {
   def input = resources.text.fromUri(getClass().getResource('...'))
   def output = layout.buildDirectory.file('...')

   inputs.property 'input', input.asString()
   outputs.file output withPropertyName 'output'

   doLast {
      Files.copy(input.asFile('UTF-8').toPath(), output.get().asFile.toPath())
   }
}

   schemas.from(
         unpackFile,
         file('...')
   )
}

I still was not able to find a better work-around that works properly.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you're interested in how we try to keep the backlog in a healthy state, please read our blog post on how we refine our backlog. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.