Gradle 8.4: `clean` fails on "java.io.IOException: Unable to delete directory..."

ianbrandt commented 11 months ago

Current Behavior

If I upgrade my project to 8.4, with no other changes, clean fails on one or more "java.io.IOException: Unable to delete directory..." errors, for example:

Execution failed for task ':my-module:my-submodule:clean'.
> java.io.IOException: Unable to delete directory 'C:\Dev\myproject\my-module\my-submodule\build'
    Failed to delete some children. This might happen because a process has files open or has its working directory set in the target directory.
    - C:\Dev\myproject\my-module\my-submodule\build\libs\my-module.my-submodule.jar
    - C:\Dev\myproject\my-module\my-submodule\build\libs

Troubleshooting details:

If I downgrade back to 8.3, with no other changes, clean succeeds every time.
In general, 1-7 out of ~300 submodules fail. Which and how many seems to depend on what was previously built.
The issue reproduces for me every time, including if I run clean again after a failure.
The same submodules seem to fail every time. I can find nothing unique about those submodules or their build logic.
The issue reproduces every time for a teammate that checked out the same revision of the project.
The issue reproduces from command line with IntelliJ completely closed, and when run from IntelliJ.
I've reproduced the issue in a brand new Windows 10 VM install with the Windows Defender Antivirus disabled and no extraneous software installed.
It still fails if I change org.gradle.parallel to false.
It still fails if I then change org.gradle.configureondemand to false.
It still fails if I then change org.gradle.configuration-cache to false.
It still fails if I then change org.gradle.caching to false.
It fails if I run gradlew --no-daemon clean.
It succeeds if I run gradlew --stop first, and then gradlew clean.
git clean -xfd fails to remove the build output referenced in the clean errors, but git clean -xfd succeeds if I run gradlew --stop first.
Searches immediately after a failure for "CPU > Associated Handles" in Resource Monitor return nothing for the files that are being reported as open by clean. Based on the above, I would have expected the Gradle Daemon process to show up with open handles to the files in question.

Expected Behavior

Running clean succeeds with Gradle 8.4, as it always does with 8.3.

Context (optional)

This blocks me from upgrading my project to Gradle 8.4.

Steps to Reproduce

We've been trying, but so far my team has been unable to create a minimal reproducer for the issue, and we're not allowed to share the project that manifests the issue. I have a couple multi-module projects with similar configurations and build logic up on GitHub, but they don't reproduce the issue. I'm happy to perform any suggested troubleshooting steps or targeted debugging.

Gradle version

8.4

Build scan URL (optional)

I'm not allowed to run build scans on this project due to security constraints.

Your Environment (optional)

Multiple Windows 10 21H2 and 22H2 systems and VMs. OpenJDK 11.0.21 2023-10-17

cobexer commented 11 months ago

Thank you for your interest in Gradle!

This issue needs a decision from the team responsible for that area. They have been informed. Response time may vary.

This might be tied to persistent compiler daemons @gradle/bt-jvm could you take a look and check if this qualifies as a regression?

big-guy commented 11 months ago

@ianbrandt A few questions...

The issue reproduces for me every time, including if I run clean again after a failure.

When you say "failure" here do you mean after a compilation failure or after the failure with running clean?

The same submodules seem to fail every time. I can find nothing unique about those submodules or their build logic

Do these submodules have annotation processors?

Are these just plain Java projects?

Are you trying to compile with a different compiler (like ecj)?

Does something like this reproduce the problem for you?

gradlew --stop
gradlew assemble
gradlew assemble --rerun-tasks
gradlew clean

ianbrandt commented 11 months ago

By failure there I just meant of an immediately previous invocation of clean, i.e. it fails repeatedly:

gradlew clean
gradlew clean

All the other tasks I've tried so far (build, test, installDist) succeed.

I don't have any annotationProcessor or kapt dependencies anywhere in my project. I am using KSP, but not in any of the modules that are failing to clean.

My project convention precompiled script plugin applies the kotlin("jvm") plugin, and almost all my modules apply the convention plugin, including all the ones that are failing. My Kotlin Gradle Plugin version is currently 1.9.0.

I am using the default Java compiler.

Running those exact commands in order does reproduce the failure of clean.

I've got VisualVM up, and the only Java processes I see staying alive after a build are org.gradle.launcher.daemon.bootstrap.GradleDaemon and org.jetbrains.kotlin.daemon.KotlinCompileDaemon. With the new persistent compiler daemon feature in 8.4, I would have expected a worker.org.gradle.process.internal.worker.GradleWorkerMain or some JavaCompileDaemon process to remain alive. Just an additional FYI.

big-guy commented 10 months ago

I would have expected a worker.org.gradle.process.internal.worker.GradleWorkerMain or some JavaCompileDaemon process to remain alive. Just an additional FYI.

Yeah, you should see a GradleWorkerMain if there are Java compiler daemons.

The bundled version of Kotlin changed from 8.3 to 8.4 (1.9.0 to 1.9.10).

If you stop the KotlinCompileDaemon, does that let you run clean?

ianbrandt commented 10 months ago

Killing the KotlinCompileDaemon process does let clean succeed.

Here's what I did:

gradlew --stop
gradlew clean # Succeeded
gradlew classes --no-build-cache --rerun-tasks
gradlew clean # Failed
# Killed KotlinCompileDaemon
gradlew clean # Succeeded

Upgrading to Kotlin 1.9.10 and KSP 1.9.10-1.0.13 seems to fix the issue:

# Updated Kotlin and KSP versions in libs.versions.toml
gradlew --stop
gradlew clean # Succeeded
gradlew classes --no-build-cache --rerun-tasks
gradlew clean # Succeeded

Oddly enough, I don't see any GradleWorkerMain processes during compilation (lots spin up when I run tests):

big-guy commented 10 months ago

Hmm, so that seems to mean it's something in the Kotlin plugin. I see similar bug reports on their tracker: https://youtrack.jetbrains.com/issue/KT-50545/IOException-Unable-to-delete-directory-Anvil-compiler-extension-fails-gradlew-clean-on-Windows-only

After you upgraded to 1.9.10, did you confirm that there weren't any KotlinCompileDaemons running and then ran the same reproducer steps?

gradlew --stop
gradlew assemble
gradlew assemble --rerun-tasks
gradlew clean

big-guy commented 10 months ago

Oddly enough, I don't see any GradleWorkerMain processes during compilation (lots spin up when I run tests):

GradleWorkerMain processes are only started if you're doing Java/Groovy/Scala compilation or using the worker API (Checkstyle, CodeNarc, etc). They're stopped at the end of the build in most cases. Only Java compilation is persistent across builds.

Kotlin has its own way of starting compiler daemons. By default, they're persistent across builds.

Gradle also starts GradleWorkerMain processes when running tests, so that's why you'd see processes then. These are stopped at the end of the task.

ianbrandt commented 10 months ago

Well shoot, when rerunning those exact steps with Kotlin 1.9.10, clean does fail again.

I did confirm there were no KotlinCompileDaemon processes after upgrade (gradlew --stop seems to kill it every time).

I'll try 1.9.20 with the assemble steps and report back.

I apply the Kotlin plugin in my project convention with the understanding that it in turn applies the Java Library Plugin. Roughly 80-90% of my codebase is Java, and I see javaCompile tasks executing in the output when running classes and assemble, but no GradleWorkerMain processes at that time. Perhaps the Kotlin Gradle Plugin does more than just delegate to the Gradle Java/Java Library Plugin when compiling Java sources. I did just submit a request for improved KGP reference documentation: https://youtrack.jetbrains.com/issue/KT-63166/KGP-Improve-reference-documentation.

ianbrandt commented 10 months ago

Here is my gradle.properties, btw:

org.gradle.caching=true
org.gradle.configuration-cache=true
org.gradle.configureondemand=true
org.gradle.java.compile-classpath-packaging=true
org.gradle.jvmargs=-Xmx10g -Dfile.encoding=UTF-8
org.gradle.parallel=true
dependency.analysis.autoapply=false
dependency.analysis.print.build.health=true

big-guy commented 10 months ago

Just a sanity check.. maybe try the reproducer steps with 8.3 too?

ianbrandt commented 10 months ago

Reproducer steps fail on clean with 8.4 and 1.9.20.

Downgraded to 8.3. Left Kotlin at 1.9.20. There was still a GradleDaemon and KotlinCompileDaemon running after gradlew --stop (presumably from the 8.4 instance), so I manually killed those. Then I tried the reproducer steps. Succeeded on clean.

ianbrandt commented 10 months ago

Looking into this further, I think this may be related to my use of the KSP Kotlin compiler plugin, similar to https://youtrack.jetbrains.com/issue/KT-50545/IOException-Unable-to-delete-directory-Anvil-compiler-extension-fails-gradlew-clean-on-Windows-only.

After the assemble --rerun-tasks step, I see now in Resource Monitor that the KotlinCompileDaemon process is hanging on to references to each of the seven JARs that are failing to clean. It's also hanging on to JARs from three subprojects related to my use of KSP. For whatever reason, those three are not being reported as subprojects that fail to clean, but one of them contains my KSP SymbolProcessor implementation. The other two, and the seven that are failing to clean, are all transitive dependencies of the SymbolProcessor project. Elsewhere in my build I apply the com.google.devtools.ksp plugin, and include the processor project as a ksp dependency.

The interesting part is, if I downgrade back to Gradle 8.3 (still Kotlin 1.9.20 and KSP 1.9.20-1.0.14), repeat the steps, and check the KotlinCompileDaemon's associated handles in Resource Monitor, I don't see it holding on to the KSP processor project JAR, or any of its nine dependency JARs. Running clean succeeds as before.

ianbrandt commented 10 months ago

Given https://docs.gradle.org/8.5-rc-1/release-notes.html#better-diagnostics-when-unable-to-delete-files, I tested with 8.5 RC1. No change, and I don't notice any particular change to the output.

I haven't gotten an exact time for it, but I am observing in Resource Monitor that if I wait long enough after the assemble --rerun-tasks step, say 5-10+ min., the running KotlinCompileDaemon does eventually release its handles to the JAR files, and clean will succeed.

ianbrandt commented 10 months ago

I've managed a shareable reproducer:

https://github.com/sdkotlin/sd-kotlin-talks/tree/af1c423a2aa59a2b0230e633c0afa6989f8254f0

PS C:\Dev\Repos\SDKotlin\sd-kotlin-talks> .\gradlew --stop
Stopping Daemon(s)
1 Daemon stopped

PS C:\Dev\Repos\SDKotlin\sd-kotlin-talks> .\gradlew assemble
Starting a Gradle Daemon, 4 stopped Daemons could not be reused, use --status for details
Configuration on demand is an incubating feature.
Calculating task graph as configuration cache cannot be reused because file 'gradle.properties' has changed.
Type-safe project accessors is an incubating feature.
...
BUILD SUCCESSFUL in 1m
46 actionable tasks: 37 executed, 9 up-to-date
Configuration cache entry stored.

PS C:\Dev\Repos\SDKotlin\sd-kotlin-talks> .\gradlew assemble --rerun-tasks
...
BUILD SUCCESSFUL in 10s
37 actionable tasks: 37 executed
Configuration cache entry reused.

PS C:\Dev\Repos\SDKotlin\sd-kotlin-talks> .\gradlew clean
Configuration on demand is an incubating feature.
Calculating task graph as configuration cache cannot be reused because file 'gradle.properties' has changed.
Type-safe project accessors is an incubating feature.
> Task :ksp-builder-generator:api:builder:clean FAILED
> Task :ksp-builder-generator:api:annotations:clean FAILED
> Task :ksp-builder-generator:processor:clean FAILED

FAILURE: Build completed with 3 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':ksp-builder-generator:api:builder:clean'.
> java.io.IOException: Unable to delete directory 'C:\Dev\Repos\SDKotlin\sd-kotlin-talks\ksp-builder-generator\api\builder\build'
    Failed to delete some children. This might happen because a process has files open or has its working directory set in the target directory.
    - C:\Dev\Repos\SDKotlin\sd-kotlin-talks\ksp-builder-generator\api\builder\build\libs\builder-1.0-SNAPSHOT.jar
    - C:\Dev\Repos\SDKotlin\sd-kotlin-talks\ksp-builder-generator\api\builder\build\libs

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':ksp-builder-generator:api:annotations:clean'.
> java.io.IOException: Unable to delete directory 'C:\Dev\Repos\SDKotlin\sd-kotlin-talks\ksp-builder-generator\api\annotations\build'
    Failed to delete some children. This might happen because a process has files open or has its working directory set in the target directory.
    - C:\Dev\Repos\SDKotlin\sd-kotlin-talks\ksp-builder-generator\api\annotations\build\libs\annotations-1.0-SNAPSHOT.jar
    - C:\Dev\Repos\SDKotlin\sd-kotlin-talks\ksp-builder-generator\api\annotations\build\libs

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.
==============================================================================

3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':ksp-builder-generator:processor:clean'.
> java.io.IOException: Unable to delete directory 'C:\Dev\Repos\SDKotlin\sd-kotlin-talks\ksp-builder-generator\processor\build'
    Failed to delete some children. This might happen because a process has files open or has its working directory set in the target directory.
    - C:\Dev\Repos\SDKotlin\sd-kotlin-talks\ksp-builder-generator\processor\build\libs\processor-1.0-SNAPSHOT.jar
    - C:\Dev\Repos\SDKotlin\sd-kotlin-talks\ksp-builder-generator\processor\build\libs

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.
==============================================================================

BUILD FAILED in 29s

It only reproduces for me on Windows, which is the only OS my internal project is built on.

One caveat, the issue also reproduces for me with this project if I downgrade to 8.3:

https://github.com/sdkotlin/sd-kotlin-talks/tree/fb17df639c08ac291470f24896ba0640d3699f32

ianbrandt commented 10 months ago

Reproduces for me with Gradle 8.2.1 as well:

https://github.com/sdkotlin/sd-kotlin-talks/tree/4ee97a9cfbc1284f951e54ebdd5e6ba6b5052b51

One difference I can think of is that I'm using Java 17 (to launch Gradle, and in my JVM Toolchains configuration) for the reproducer project, whereas my internal project is still all on Java 11.

The only other difference I can think of is my KSP SymbolProcessor in my internal project is somewhat more involved, and has dependencies on more subprojects than the reproducer does.

Otherwise, I'm not sure why my internal project only manifests the issue with Gradle 8.4+, and not Gradle 8.3.

Let me know if I should pursue this upstream with Kotlin, or if it makes sense to keep this open as a potential Gradle issue.

big-guy commented 10 months ago

@ianbrandt thank you for digging more into this. Can you open an issue upstream with Kotlin? I'll raise this with them too.

Now that we have a reproducer, we can take a look if we somehow changed something in 8.4 that made this more likely to happen, but it looks like it could be an issue for Kotlin to solve.

ianbrandt commented 10 months ago

Thanks, @big-guy. I added to the open issue here: https://youtrack.jetbrains.com/issue/KT-50545/IOException-Unable-to-delete-directory-Anvil-compiler-extension-fails-gradlew-clean-on-Windows-only#focus=Comments-27-8371337.0-0.

jp196 commented 9 months ago

I am also seeing the same symptoms with clean failing 100% of the time on Gradle 8.4 and 8.5. Reverting to Gradle 8.3 fixes it 100% of the time.

I've tried stopping the Gradle daemon, then running clean. It doesn't help.

We have only Groovy scripts - no Kotlin (although I don't know if a plugin may be using Kotlin, or whether the Kotlin engine could be invoked for Groovy scripts).

The specific output file that it is failing to clean is an *.exe file that is created by a plugin we are using called "The Badass Runtime Plugin" (https://badass-runtime-plugin.beryx.org/releases/latest/) which uses jpackage to create an installer application. Jpackage is using the WIX toolkit to create the executable. I would entertain the notion that it could be the plugin, jpackage, or WIX except that Gradle 8.4 consistently doesn't clean it, whereas 8.3 consistently does.

Using Java 17.0.7, Windows 11.

[UPDATE] So the EXE file that it is not deleting has the "Read-only" attribute set to true (checked). This is true if the build is performed using Gradle 8.4 or 8.3. However, 8.3 happily deletes the file, whereas 8.4 does not. I'm not sure whether this is a "bug" or a "safety feature". There does not seem to be a process holding the file open, so I assume that it is the handling of the read-only attribute that differs.

Hopefully this will be of use to other people who may stumble across this comment.

ianbrandt commented 9 months ago

I can confirm that on Windows 10 if I check "Properties > Attributes > Read-only" for a file under "build/", and run:

gradlew --stop
gradlew clean

With 8.3 it succeeds, but with 8.4 and 8.5 it fails:

Unable to delete directory 'C:\Dev\Repos\SDKotlin\sd-kotlin-talks\tdd-in-kotlin\build'
  Failed to delete some children. This might happen because a process has files open or has its working directory set in the target directory.
  - C:\Dev\Repos\SDKotlin\sd-kotlin-talks\tdd-in-kotlin\build\libs\tdd-in-kotlin-1.0-SNAPSHOT.jar
  - C:\Dev\Repos\SDKotlin\sd-kotlin-talks\tdd-in-kotlin\build\libs

If I then uncheck "Read-only", clean succeeds.

Reproducers:

While this may be a different trigger than my KotlinCompileDaemon holding open file references issue, I wonder if it in any way relates to the "worsening" of that issue I experience when upgrading past 8.3. (I did just confirm that the build output files the KotlinCompileDaemon is hanging on to aren't set to read only.)

I'm not a Windows expert, but I believe the "Read-only" attribute is intended to signify that a file shouldn't be modified, not that it shouldn't be deleted from a folder. If so, I'd say the 8.3 behavior was correct.

ov7a commented 9 months ago

As @chschu pointed out here, this issue is caused by

https://github.com/gradle/gradle/pull/26011

oleg-nenashev commented 8 months ago

Returning to the comment from @chschu

The File.delete() previously used will delete a file even if it is read-only, while Files.deleteIfExists(Path) throws a java.nio.file.AccessDeniedException in that case.

To be more specific about that, the default Java setup allows deleting read-only files in Files.deleteIfExists() with a default security manager. So most likely this is a bug in the custom security manager implementation. I will take a reproducer and try to debug it

oleg-nenashev commented 7 months ago

Quick update on that file locking support ticket:

Last Friday I set up Gradle 8.6 and the reproducer on my local Windows machine (amd64 / Windows Pro latest, I do not remember what is the FS)
Was unable to reproduce with Java 17 and my FS
Was about down grading to 8.4, but got network issues and just failing wrapper
Got I'll :(

I set up the same stuff on my MacBook, and presumably there I will be able to check the open handles there too

oleg-nenashev commented 7 months ago

FWIW https://github.com/gradle/gradle/pull/27392 was merged only towards 8.7-RC-1, so I do have doubts the issue is fully resolved

oleg-nenashev commented 7 months ago

NOTE: https://github.com/gradle/gradle/commit/2951eca66530de10f418c4dfd1a43c460aad08bf also added proper error propagation, but we can conclude this issue without a new trace

AsfhtgkDavid commented 6 months ago

Hi there. I have similar problem with delete jpackage dir. The problem is observed with java 21, gradle 8.6 and 8.7-rc-2.

ov7a commented 6 months ago

@AsfhtgkDavid Please open a new issue with a MWE.

KatharinaJenID commented 5 months ago

I had the same problem, but it works after upgrading zu Gradle 8.7.

gzahavi commented 4 months ago

ran into a similar problem in gradle 8.1.1 for whatever reason, clean would work once, when i ran the 'configure project' step first, for example via gradlew tasks --dry-run https://discuss.gradle.org/t/how-to-run-only-configuration-phase-from-command-line/35169/5

gradle / gradle