search

google / ksp

Kotlin Symbol Processing API
https://github.com/google/ksp
Apache License 2.0
2.75k stars 258 forks source link

KSP is not thread safe in a multiplex worker environment #1385

Open Bencodes opened 1 year ago

Bencodes commented 1 year ago

When using KSP in a persistent multiplexed worker environment we see unexplainable build errors claiming that KSP is unable to resolve types defined in the same module that's currently being processed. We believe this is happening because of the changed introduced here to avoid leaking memory, are now causing KSP to wipe out the resolver cached needed by other threads.

A few examples of the errors that we are seeing:

KSTypeReference#resolve returning null:

error: [ksp] java.lang.NullPointerException
        at com.google.devtools.ksp.symbol.impl.kotlin.KSTypeReferenceImpl.resolve(KSTypeReferenceImpl.kt:124)
        at com.lyft.ksp.skabbard.KspUtilsKt.resolveToUnderlying(KspUtils.kt:61)
        at com.lyft.ksp.skabbard.ClassOnlyTraversingResolver$getSymbolsWithAnnotation$1.invoke(ClassOnlyTraversingResolver.kt:37)
        at com.lyft.ksp.skabbard.ClassOnlyTraversingResolver$getSymbolsWithAnnotation$1.invoke(ClassOnlyTraversingResolver.kt:33)
        at kotlin.sequences.FilteringSequence$iterator$1.calcNext(Sequences.kt:171)
        at kotlin.sequences.FilteringSequence$iterator$1.hasNext(Sequences.kt:194)
        at kotlin.sequences.FilteringSequence$iterator$1.calcNext(Sequences.kt:169)
        at kotlin.sequences.FilteringSequence$iterator$1.hasNext(Sequences.kt:194)
        at kotlin.sequences.TransformingSequence$iterator$1.hasNext(Sequences.kt:214)

KSP is unable to resolve types defined in the same module that's being processed:

error: [ksp] java.lang.IllegalStateException: Cannot find package fragment com.lyft.android.scoop.unidirectional.interop
vFile = /private/var/tmp/_bazel_blee/499a001013731d09bffd82f8601a3161/execroot/instantandroid/instant-scoop/unidirectional/interop/src/main/java/com/lyft/android/scoop/unidirectional/interop/ScoopInteropGraph.kt, file package = 'com.lyft.android.scoop.unidirectional.interop'
        at org.jetbrains.kotlin.resolve.lazy.declarations.AbstractDeclarationProviderFactory.diagnoseMissingPackageFragment(AbstractDeclarationProviderFactory.java:51)
        at org.jetbrains.kotlin.resolve.lazy.ResolveSession.getPackageFragmentOrDiagnoseFailure(ResolveSession.java:275)
        at org.jetbrains.kotlin.resolve.lazy.LazyDeclarationResolver.getMemberScopeDeclaredIn$frontend(LazyDeclarationResolver.kt:223)
        at org.jetbrains.kotlin.resolve.lazy.LazyDeclarationResolver.findClassDescriptorIfAny(LazyDeclarationResolver.kt:71)
        at org.jetbrains.kotlin.resolve.lazy.LazyDeclarationResolver.getClassDescriptorIfAny(LazyDeclarationResolver.kt:59)
        at org.jetbrains.kotlin.resolve.lazy.LazyDeclarationResolver$resolveToDescriptor$1.visitClass(LazyDeclarationResolver.kt:100)
        at org.jetbrains.kotlin.resolve.lazy.LazyDeclarationResolver$resolveToDescriptor$1.visitClass(LazyDeclarationResolver.kt:94)
ZacSweers commented 4 months ago

This is a pretty bazel-specific use case, and the result of bazel running every KSP processor in its own thread rather than the cooperative round-based approach that KSP is implemented as. I don't see a path forward on this in the way it's currently implemented in rules_kotlin. Correct me if I'm misunderstanding @Bencodes