search

gradle / gradle

Adaptable, fast automation for all
https://gradle.org
Apache License 2.0
17.02k stars 4.78k forks source link

Compilations are regularly out of date due to non-deterministic classpath ordering #16682

Open ZacSweers opened 3 years ago

ZacSweers commented 3 years ago

Expected Behavior

Classpath order should be consistent or irrelevant when containing unique elements

Current Behavior

We regularly see Kotlin and Java compilations (both of which extend AbstractCompile) re-run due to dependencies on their classpath property changing order. This is a big drain on productivity and there doesn't seem to be any rhyme or reason. We can see why they re-ran in build scans.

This is one example, where the only change was to a test source file yet several projects recompiled.


The task was not up-to-date because of the following reasons: |  
-- | --
Input property 'classpath' file /Users/zsweers/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-parcelize-runtime/1.4.32/8ba46fd36616d05f01881df92ee389029ad3f823/kotlin-parcelize-runtime-1.4.32.jar has been removed. |  
Input property 'classpath' file /Users/zsweers/.gradle/caches/modules-2/files-2.1/slack.internal/slack-lint-annotations/00003/241d2cec0576e17eef9cb4a5419273f8f9ab1e8f/slack-lint-annotations-00003.jar has been added. |  
Input property 'classpath' file /Users/zsweers/.gradle/caches/modules-2/files-2.1/com.jakewharton.rx3/replaying-share-kotlin/3.0.0/af8b55f3f0a03bdd7934550ae0e4af77de886c7f/replaying-share-kotlin-3.0.0.jar has been removed. |  

Context

See above

Steps to Reproduce

I'm not able to decipher what causes this in gradle, but suspect it's due to the combination of the classpath being order-sensative and gradle's underlying components operating in parallel when adding dependency jars to the classpath property.

Your Environment

Gradle 6.8.3, macOS 11.2.3

runningcode commented 3 years ago

Could this be related to? https://github.com/gradle/gradle/issues/15626

ljacomet commented 3 years ago

The @Classpath annotation makes order relevant because of the shadowing property of a classpath on the JVM: files found first will hide ones found later.

How are the classpath of the tasks composed? Dependency resolution only or some form of FileCollection merging on top?

or irrelevant when containing unique elements

Can you clarify what you mean with the above? The order of elements in a list of one element should never be an issue.

ZacSweers commented 3 years ago

I don't know how they're composed, most of this is wholly opaque to consumers. We see it in both KotlinCompile and JavaCompile tasks

image image

It reads to me like gradle has non-deterministic ordering of inputs on the classpath property and should offer a means of sorting them to make them deterministic. Between this issue and the related #15626, there doesn't seem to be a reliable way to trace where these are coming from and when.

or irrelevant when containing unique elements

This might be a technical infeasibility, but if they jars contain distinct classes then I'd expect it not to matter what order they're in. If I have lib1.jar with classes A, B, and C and lib2.jar with classes D, E, and F, why does it matter what order the jars are in?

ljacomet commented 3 years ago

I am sorry but in the screenshot provided I do not see the ordering difference. Was there more content below what was shared?

> or irrelevant when containing unique elements

This might be a technical infeasibility, but if they jars contain distinct classes then I'd expect it not to matter what order they're in. If I have lib1.jar with classes A, B, and C and lib2.jar with classes D, E, and F, why does it matter what order the jars are in?

My bad, I misunderstood "unique". I see what you are suggesting but I fear that would be quite a cost as instead of only snapshotting JARs, Gradle would need to also identify any class overlap.

ZacSweers commented 3 years ago

AGP does a post-processing task check for duplicate classes that can run in parallel, would that be a reasonable compromise?

https://android.googlesource.com/platform/tools/base/+/studio-master-dev/build-system/gradle-core/src/main/java/com/android/build/gradle/internal/tasks/CheckDuplicateClassesTask.kt

ZacSweers commented 3 years ago

Sorry I missed the first part of that

I am sorry but in the screenshot provided I do not see the ordering difference. Was there more content below what was shared?

I can try to get more but it's a large output and Gradle Enterprise doesn't always render it well nor offer an easy way to share it. Here's another text-only output


The task was not up-to-date because of the following reasons: |  
-- | --
Input property 'classpath' file /Users/zsweers/.gradle/caches/modules-2/files-2.1/slack.internal/jsr250/00001/73264bd42585df63ad72dd8af4cc1bd2e39aa873/jsr250-00001.jar has been removed. |  
Input property 'classpath' file /Users/zsweers/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-core-jvm/1.4.3/1051e11eb789ed1df93412bebe0f06e9d29160e4/kotlinx-coroutines-core-jvm-1.4.3.jar has been added. |  
Input property 'classpath' file /Users/zsweers/.gradle/caches/transforms-3/644465c02ac6678e96cf29655143b779/transformed/appcompat-1.2.0-api.jar has been removed.
In build A In build B
libraries/app-lifecycle/build/intermediates/compile_r_class_jar/release/R.jar libraries/app-lifecycle/build/intermediates/compile_r_class_jar/release/R.jar
libraries/model/build/intermediates/compile_library_classes_jar/release/classes.jar libraries/model/build/intermediates/compile_library_classes_jar/release/classes.jar
libraries/foundation/slack-commons/build/intermediates/compile_library_classes_jar/release/classes.jar libraries/foundation/slack-commons/build/intermediates/compile_library_classes_jar/release/classes.jar
libraries/foundation/feature-flag/build/intermediates/compile_library_classes_jar/release/classes.jar libraries/foundation/feature-flag/build/intermediates/compile_library_classes_jar/release/classes.jar
libraries/foundation/minimized-feature-flag-library/build/libs/minimized-feature-flag-library.jar libraries/foundation/minimized-feature-flag-library/build/libs/minimized-feature-flag-library.jar
libraries/foundation/slack-di/build/libs/slack-di.jar libraries/foundation/slack-di/build/libs/slack-di.jar
libraries/telemetry/telemetry-public/build/intermediates/compile_library_classes_jar/release/classes.jar libraries/telemetry/telemetry-public/build/intermediates/compile_library_classes_jar/release/classes.jar
libraries/foundation/coroutines/build/libs/coroutines.jar libraries/foundation/coroutines/build/libs/coroutines.jar
services/pending-actions-model/build/libs/pending-actions-model.jar services/pending-actions-model/build/libs/pending-actions-model.jar
libraries/foundation/feature-flag-library/build/libs/feature-flag-library.jar libraries/foundation/feature-flag-library/build/libs/feature-flag-library.jar
slack.internal:slack-lint-annotations:00003 (slack-lint-annotations-00003.jar) slack.internal:slack-lint-annotations:00003 (slack-lint-annotations-00003.jar)
com.squareup.anvil:annotations:2.2.1 (annotations-2.2.1.jar) com.squareup.anvil:annotations:2.2.1 (annotations-2.2.1.jar)
org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.4.32 (kotlin-stdlib-jdk8-1.4.32.jar) org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.4.32 (kotlin-stdlib-jdk8-1.4.32.jar)
slack.internal:jsr250:00001 (jsr250-00001.jar) org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.4.3 (kotlinx-coroutines-core-jvm-1.4.3.jar)
GRADLE USER HOME/caches/transforms-3/644465c02ac6678e96cf29655143b779/transformed/appcompat-1.2.0-api.jar org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.4.32 (kotlin-stdlib-jdk7-1.4.32.jar)
GRADLE USER HOME/caches/transforms-3/369e4e0bf276510db013e8e90adb2793/transformed/fragment-1.3.3-api.jar com.slack.data:client-thrifty:0.0.295 (client-thrifty-0.0.295.jar)
GRADLE USER HOME/caches/transforms-3/2787dbf14e437d15b5de93a4aa55f70f/transformed/appcompat-resources-1.2.0-api.jar com.microsoft.thrifty:thrifty-runtime:0.4.3 (thrifty-runtime-0.4.3.jar)
GRADLE USER HOME/caches/transforms-3/0a671bab96f86a4201612051cc10f12c/transformed/drawerlayout-1.1.1-api.jar com.squareup.okio:okio:2.10.0 (okio-jvm-2.10.0.jar)
GRADLE USER HOME/caches/transforms-3/67bd3dbb232c5191a4d954919d81523d/transformed/viewpager-1.0.0-api.jar org.jetbrains.kotlin:kotlin-stdlib:1.4.32 (kotlin-stdlib-1.4.32.jar)
GRADLE USER HOME/caches/transforms-3/ca86c56f030f18f4a2693fa3b77052d8/transformed/loader-1.1.0-api.jar slack.internal:jsr250:00001 (jsr250-00001.jar)
GRADLE USER HOME/caches/transforms-3/a45e3847cfc19e64f60a15a9f1859b2a/transformed/activity-1.2.2-api.jar GRADLE USER HOME/caches/transforms-3/644465c02ac6678e96cf29655143b779/transformed/appcompat-1.2.0-api.jar
GRADLE USER HOME/caches/transforms-3/411b20fae7c907ca10ee6870d4a7d201/transformed/vectordrawable-animated-1.1.0-api.jar GRADLE USER HOME/caches/transforms-3/369e4e0bf276510db013e8e90adb2793/transformed/fragment-1.3.3-api.jar
GRADLE USER HOME/caches/transforms-3/23f3ad5de7559aeadef14dcea6180a02/transformed/vectordrawable-1.1.0-api.jar GRADLE USER HOME/caches/transforms-3/2787dbf14e437d15b5de93a4aa55f70f/transformed/appcompat-resources-1.2.0-api.jar
GRADLE USER HOME/caches/transforms-3/7b3a2e2cdad3729a5a455077d72dc629/transformed/customview-1.1.0-api.jar GRADLE USER HOME/caches/transforms-3/0a671bab96f86a4201612051cc10f12c/transformed/drawerlayout-1.1.1-api.jar
GRADLE USER HOME/caches/transforms-3/0a2ab9d9e4ebcb9b643299034d72d280/transformed/core-1.5.0-rc01-api.jar GRADLE USER HOME/caches/transforms-3/67bd3dbb232c5191a4d954919d81523d/transformed/viewpager-1.0.0-api.jar
GRADLE USER HOME/caches/transforms-3/7e0a35a469f236d9ecf7f4f14b6cf69f/transformed/cursoradapter-1.0.0-api.jar GRADLE USER HOME/caches/transforms-3/ca86c56f030f18f4a2693fa3b77052d8/transformed/loader-1.1.0-api.jar
GRADLE USER HOME/caches/transforms-3/7cd65c51342237d8233ad55a775f76ef/transformed/lifecycle-runtime-2.3.1-api.jar GRADLE USER HOME/caches/transforms-3/a45e3847cfc19e64f60a15a9f1859b2a/transformed/activity-1.2.2-api.jar
GRADLE USER HOME/caches/transforms-3/f44468022ba7d9ad6bea233af154cf2a/transformed/versionedparcelable-1.1.1-api.jar GRADLE USER HOME/caches/transforms-3/411b20fae7c907ca10ee6870d4a7d201/transformed/vectordrawable-animated-1.1.0-api.jar
androidx.collection:collection:1.1.0 (collection-1.1.0.jar) GRADLE USER HOME/caches/transforms-3/23f3ad5de7559aeadef14dcea6180a02/transformed/vectordrawable-1.1.0-api.jar
GRADLE USER HOME/caches/transforms-3/0bb8c26b886d25b0c19b32abe80785aa/transformed/lifecycle-viewmodel-savedstate-2.3.1-api.jar GRADLE USER HOME/caches/transforms-3/7b3a2e2cdad3729a5a455077d72dc629/transformed/customview-1.1.0-api.jar
GRADLE USER HOME/caches/transforms-3/61176ee706964195172ec68f5cbe605b/transformed/lifecycle-viewmodel-2.3.1-api.jar GRADLE USER HOME/caches/transforms-3/0a2ab9d9e4ebcb9b643299034d72d280/transformed/core-1.5.0-rc01-api.jar
GRADLE USER HOME/caches/transforms-3/30e62fc875692e30f6d07903760858dd/transformed/savedstate-1.1.0-api.jar GRADLE USER HOME/caches/transforms-3/7e0a35a469f236d9ecf7f4f14b6cf69f/transformed/cursoradapter-1.0.0-api.jar
GRADLE USER HOME/caches/transforms-3/550b1e32776adcdf62a7a02fa9991358/transformed/lifecycle-livedata-core-2.3.1-api.jar GRADLE USER HOME/caches/transforms-3/7cd65c51342237d8233ad55a775f76ef/transformed/lifecycle-runtime-2.3.1-api.jar
androidx.lifecycle:lifecycle-common:2.3.1 (lifecycle-common-2.3.1.jar) GRADLE USER HOME/caches/transforms-3/f44468022ba7d9ad6bea233af154cf2a/transformed/versionedparcelable-1.1.1-api.jar
androidx.arch.core:core-common:2.1.0 (core-common-2.1.0.jar) androidx.collection:collection:1.1.0 (collection-1.1.0.jar)
GRADLE USER HOME/caches/transforms-3/debe55c4fb5990782c8c7d14fe731962/transformed/interpolator-1.0.0-api.jar GRADLE USER HOME/caches/transforms-3/0bb8c26b886d25b0c19b32abe80785aa/transformed/lifecycle-viewmodel-savedstate-2.3.1-api.jar
androidx.annotation:annotation:1.2.0 (annotation-1.2.0.jar) GRADLE USER HOME/caches/transforms-3/61176ee706964195172ec68f5cbe605b/transformed/lifecycle-viewmodel-2.3.1-api.jar
com.google.guava:guava:30.1.1-android (guava-30.1.1-android.jar) GRADLE USER HOME/caches/transforms-3/30e62fc875692e30f6d07903760858dd/transformed/savedstate-1.1.0-api.jar
com.slack.data:client-thrifty:0.0.295 (client-thrifty-0.0.295.jar) GRADLE USER HOME/caches/transforms-3/550b1e32776adcdf62a7a02fa9991358/transformed/lifecycle-livedata-core-2.3.1-api.jar
com.jakewharton.rxrelay3:rxrelay:3.0.0 (rxrelay-3.0.0.jar) androidx.lifecycle:lifecycle-common:2.3.1 (lifecycle-common-2.3.1.jar)
io.reactivex.rxjava3:rxjava:3.0.12 (rxjava-3.0.12.jar) androidx.arch.core:core-common:2.1.0 (core-common-2.1.0.jar)
GRADLE USER HOME/caches/transforms-3/f9e703c3301cbac0c6a0b99427333530/transformed/timber-4.7.1-api.jar GRADLE USER HOME/caches/transforms-3/debe55c4fb5990782c8c7d14fe731962/transformed/interpolator-1.0.0-api.jar
com.google.dagger:dagger:2.35 (dagger-2.35.jar) androidx.annotation:annotation:1.2.0 (annotation-1.2.0.jar)
javax.inject:javax.inject:1 (javax.inject-1.jar) com.google.guava:guava:30.1.1-android (guava-30.1.1-android.jar)
org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.4.32 (kotlin-stdlib-jdk7-1.4.32.jar) com.jakewharton.rxrelay3:rxrelay:3.0.0 (rxrelay-3.0.0.jar)
org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.4.3 (kotlinx-coroutines-core-jvm-1.4.3.jar) io.reactivex.rxjava3:rxjava:3.0.12 (rxjava-3.0.12.jar)
com.microsoft.thrifty:thrifty-runtime:0.4.3 (thrifty-runtime-0.4.3.jar) GRADLE USER HOME/caches/transforms-3/f9e703c3301cbac0c6a0b99427333530/transformed/timber-4.7.1-api.jar
com.squareup.okio:okio:2.10.0 (okio-jvm-2.10.0.jar) com.google.dagger:dagger:2.35 (dagger-2.35.jar)
org.jetbrains.kotlin:kotlin-stdlib:1.4.32 (kotlin-stdlib-1.4.32.jar) javax.inject:javax.inject:1 (javax.inject-1.jar)
org.jetbrains.kotlin:kotlin-stdlib-common:1.4.32 (kotlin-stdlib-common-1.4.32.jar) org.jetbrains.kotlin:kotlin-stdlib-common:1.4.32 (kotlin-stdlib-common-1.4.32.jar)
org.jetbrains:annotations:16.0.1 (annotations-16.0.1.jar) org.jetbrains:annotations:16.0.1 (annotations-16.0.1.jar)
org.reactivestreams:reactive-streams:1.0.3 (reactive-streams-1.0.3.jar) org.reactivestreams:reactive-streams:1.0.3 (reactive-streams-1.0.3.jar)
GRADLE USER HOME/caches/transforms-3/1acd4710703ebf8e3f546dbd9f0d3add/transformed/annotation-experimental-1.0.0-api.jar GRADLE USER HOME/caches/transforms-3/1acd4710703ebf8e3f546dbd9f0d3add/transformed/annotation-experimental-1.0.0-api.jar
com.google.guava:failureaccess:1.0.1 (failureaccess-1.0.1.jar) com.google.guava:failureaccess:1.0.1 (failureaccess-1.0.1.jar)
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava (listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar) com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava (listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar)
com.google.code.findbugs:jsr305:3.0.2 (jsr305-3.0.2.jar) com.google.code.findbugs:jsr305:3.0.2 (jsr305-3.0.2.jar)
org.checkerframework:checker:3.6.0 (checker-3.6.0.jar) org.checkerframework:checker:3.6.0 (checker-3.6.0.jar)
com.google.errorprone:error_prone_annotations:2.6.0 (error_prone_annotations-2.6.0.jar) com.google.errorprone:error_prone_annotations:2.6.0 (error_prone_annotations-2.6.0.jar)
com.google.j2objc:j2objc-annotations:1.3 (j2objc-annotations-1.3.jar) com.google.j2objc:j2objc-annotations:1.3 (j2objc-annotations-1.3.jar)
/Users/zsweers/dev/android/sdk/platforms/android-30/android.jar /Users/zsweers/dev/android/sdk/platforms/android-30/android.jar
/Users/zsweers/dev/android/sdk/build-tools/29.0.2/core-lambda-stubs.jar /Users/zsweers/dev/android/sdk/build-tools/29.0.2/core-lambda-stubs.jar
ZacSweers commented 3 years ago

@ljacomet is there anything else I could check for you?

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you're interested in how we try to keep the backlog in a healthy state, please read our blog post on how we refine our backlog. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.

ZacSweers commented 2 years ago

Still a major problem for Java compilations