search

Guardsquare / proguard

ProGuard, Java optimizer and obfuscator
https://www.guardsquare.com/en/products/proguard
GNU General Public License v2.0
2.88k stars 411 forks source link

NPE while reading a jar compiled from Kotlin 1.5 #179

Open philipguin opened 3 years ago

philipguin commented 3 years ago

Hi there,

I have a Kotlin/Java project that I'm trying to obfuscate as a jar exported from IntelliJ, but I'm getting a NullPointerException during obfuscation. I'm using Kotlin 1.5.2 and targeting JDK 1.8.

Is it possible that Kotlin 1.5 isn't yet supported? I couldn't find this documented anywhere. If it isn't, then let me know, otherwise...

I previously had this working before porting the project to Kotlin, but since I'm relying on reflection, the -keepkotlinmetadata feature is something I now need. I should note that I'm not using the Gradle plugin, as I'm not currently using Gradle. Instead, I checked out the repo and built using gradlew build -x test.

I'm not keen on sharing my non-obfuscated jar, but if I could get some pointers, I'd be happy to try reproducing this with a small test project.

Basic ProGuard settings (full configuration attached: ProGuardConfig.pro.txt):

-injars ..\dist\Sojourners\bin\NebRax.jar
-outjars ..\dist\Sojourners\bin\NebRax_obf.jar

-libraryjars 'C:\Program Files\Java\jre1.8.0_202\lib\rt.jar'
-libraryjars ..\dist\Sojourners\bin\lib

-dontoptimize
-printmapping ProGuardMapping.pro
-keeppackagenames shaders,shaders.**,phil.gl.shaders,phil.gl.shaders.**
-repackageclasses ''
-keepattributes Signature,RuntimeVisibleAnnotations,RuntimeVisibleParameterAnnotations,RuntimeVisibleTypeAnnotations,AnnotationDefault
-keepparameternames
-ignorewarnings
-keepkotlinmetadata
-verbose

-keep class kotlin.Metadata {
    *;
}

...

Exception stacktrace:

ProGuard, version 7.1.2
Reading input...
Reading program jar [H:\NebRax\Exports\dist\Sojourners\bin\NebRax.jar] (filtered)
java.io.IOException: Can't read [H:\NebRax\Exports\dist\Sojourners\bin\NebRax.jar] (Can't read Kotlin module file [META-INF/NebulousRax.kotlin_module] (Error while reading Kotlin module file))
    at proguard.InputReader.readInput(InputReader.java:266)
    at proguard.InputReader.readInput(InputReader.java:230)
    at proguard.InputReader.readInput(InputReader.java:207)
    at proguard.InputReader.execute(InputReader.java:135)
    at proguard.ProGuard.readInput(ProGuard.java:310)
    at proguard.ProGuard.execute(ProGuard.java:113)
    at proguard.ProGuard.main(ProGuard.java:717)
Caused by: java.io.IOException: Can't read Kotlin module file [META-INF/NebulousRax.kotlin_module] (Error while reading Kotlin module file)
    at proguard.resources.kotlinmodule.io.KotlinModuleDataEntryReader.read(KotlinModuleDataEntryReader.java:55)
    at proguard.io.FilteredDataEntryReader.read(FilteredDataEntryReader.java:85)
    at proguard.io.FilteredDataEntryReader.read(FilteredDataEntryReader.java:85)
    at proguard.io.RenamedDataEntryReader.read(RenamedDataEntryReader.java:97)
    at proguard.io.FilteredDataEntryReader.read(FilteredDataEntryReader.java:85)
    at proguard.io.FilteredDataEntryReader.read(FilteredDataEntryReader.java:85)
    at proguard.io.JarReader.read(JarReader.java:84)
    at proguard.io.DirectorySource.readFiles(DirectorySource.java:68)
    at proguard.io.DirectorySource.pumpDataEntries(DirectorySource.java:54)
    at proguard.InputReader.readInput(InputReader.java:262)
    ... 6 more
Caused by: java.lang.RuntimeException: Error while reading Kotlin module file
    at proguard.resources.kotlinmodule.io.KotlinModuleReader.visitKotlinModule(KotlinModuleReader.java:84)
    at proguard.resources.kotlinmodule.KotlinModule.accept(KotlinModule.java:70)
    at proguard.resources.kotlinmodule.io.KotlinModuleDataEntryReader.read(KotlinModuleDataEntryReader.java:49)
    ... 15 more
Caused by: java.lang.NullPointerException
    at java.util.Objects.requireNonNull(Unknown Source)
    at proguard.resources.kotlinmodule.io.KotlinModuleReader.visitKotlinModule(KotlinModuleReader.java:65)
    ... 17 more

Thanks for your time.

mrjameshamilton commented 3 years ago

Hi @philipguin ! Thanks for the report. You are correct, Kotlin 1.5 is not supported with ProGuard 7.1.

We released 7.2.0-beta1 on Monday that now supports Kotlin 1.5. Can you try it out? https://github.com/Guardsquare/proguard/releases/tag/v7.2.0-beta1

Be aware there is already a known issue with beta1 that it might crash when executed on Java 11+, on Java 8 it should be fine.

You can instead build it yourself from the beta branch which should already fix the Java 11+ issue: https://github.com/Guardsquare/proguard/tree/beta

philipguin commented 3 years ago

@mrjameshamilton So I checked out the beta branch, built it, then ran again on my jar. This time ProGuard ran until completion and produced a jar, but spat out a bunch of new warnings. The new jar produced the same crash I got when running without -keepkotlinmetadata, probably because all the metadata was ignored, as indicated by the warnings.

Errors:

Warning: Kotlin metadata errors encountered in animation/AnimationSet. Not processing the metadata for this class.
  Type has no reference for its type alias "kotlin/collections/HashMap".
  Type has no reference for its type alias "kotlin/collections/HashMap".
Warning: Kotlin metadata errors encountered in animation/AnimationSet$animate$2. Not processing the metadata for this class.
  Function <anonymous> has no reference for its referencedMethod.
Warning: Kotlin metadata errors encountered in asset/NebRaxAssets. Not processing the metadata for this class.
  Type has no reference for its type alias "kotlin/collections/ArrayList".
Warning: Kotlin metadata errors encountered in assimp/AssimpImportScene. Not processing the metadata for this class.
  Type has no reference for its type alias "kotlin/collections/ArrayList".
Warning: Kotlin metadata errors encountered in audio/LerpLine$sortPivots$1. Not processing the metadata for this class.
  Function <anonymous> has no reference for its referencedMethod.
Warning: Kotlin metadata errors encountered in auto_resources/AutoResourceBinder. Not processing the metadata for this class.
  Type has no reference for its type alias "kotlin/collections/HashMap".
Warning: Kotlin metadata errors encountered in auto_resources/AutoResourceBinder$ClassBookkeeping. Not processing the metadata for this class.
  Type has no reference for its type alias "kotlin/collections/ArrayList".
  Type has no reference for its type alias "kotlin/collections/ArrayList".
Warning: Kotlin metadata errors encountered in auto_resources/AutoResourceBinder$Companion. Not processing the metadata for this class.
  Type has no reference for its type alias "kotlin/collections/HashSet".
  Type has no reference for its type alias "kotlin/collections/HashSet".
  Type has no reference for its type alias "kotlin/collections/HashSet".
  Type has no reference for its type alias "kotlin/collections/HashSet".
Warning: Kotlin metadata errors encountered in auto_resources/AutoResourceBinder$clearUnusedBookkeeping$1$1. Not processing the metadata for this class.
  Function <anonymous> has no reference for its referencedMethod.
Warning: Kotlin metadata errors encountered in auto_resources/AutoResourceBinder$createClassData$1$1$12. Not processing the metadata for this class.
  Function <anonymous> has no reference for its referencedMethod.
...

Seems like its stumbling on Kotlin standard library typealiases, though kotlin-stdlib.jar is definitely inside the included lib folder. The typealiases are defined in the kotlin-stdlib-sources.jar, though I suppose they would compile out being typealiases... interesting that they'd be included in the metadata in that case. Perhaps they are special cased by Kotlin reflection code reading the metadata?

EDIT:

Here's the entirety of ProGuard's log output, if it's helpful: proguard.log I suppose there could be another causative warning in there I missed!

mrjameshamilton commented 3 years ago

Hi @philipguin ! Thanks for the further information.

The typealiases are a purely Kotlin feature, so indeed they don't appear in the bytecode, and the information about them is therefore only available via the metadata. This could be important when using them for example, via reflection.

The warnings mean that ProGuard was unable to properly initialize some of the metadata. It's not uncommon to see at least some of these warnings in the log. Sometimes they are due to bugs or missing features in the initialization code, sometimes they are due to incorrect metadata. When ProGuard encounters these initialization problems it removes the metadata that's attached to the class, so that later processing steps can be sure of having valid metadata. Often the app still works fine when you have some of these warnings.

Are the Kotlin stdlib classes in H:\NebRax\Exports\dist\Sojourners\bin\lib or are they packaged as part of your app? There is a known issue that ProGuard cannot read / store Kotlin metadata from library classes (-libraryjars).

The new jar produced the same crash I got when running without -keepkotlinmetadata, probably because all the metadata was ignored, as indicated by the warnings.

What is the error that you get?

Are you able to share the project (you can send it to james [dot] hamilton [at] guardsquare [dot] com)? Or a sample that reproduces the problem?

philipguin commented 3 years ago

All depended jars are under H:\NebRax\Exports\dist\Sojourners\bin\lib, including Kotlin stdlib. I'll investigate packaging the stdlib inside the main jar (possibly along with other libs) and report back.

The error I get is from Kotlin reflection failing:

2021-07-29 22:41:39:856 [SEVERE] {main.NebRaxApp main} Unhandled exception: java.lang.IllegalStateException: Incomplete hierarchy for class DemoPointLightBehavior, unresolved classes [behavior.GameBehavior]
2021-07-29 22:41:39:858 [SEVERE] {main.NebRaxApp main} Stacktrace: java.lang.IllegalStateException: Incomplete hierarchy for class DemoPointLightBehavior, unresolved classes [behavior.GameBehavior]
    at kotlin.reflect.jvm.internal.impl.descriptors.runtime.components.RuntimeErrorReporter.reportIncompleteHierarchy(RuntimeErrorReporter.kt:26)
    at kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassTypeConstructor.computeSupertypes(DeserializedClassDescriptor.kt:201)
    at kotlin.reflect.jvm.internal.impl.types.AbstractTypeConstructor$supertypes$1.invoke(AbstractTypeConstructor.kt:83)
    at kotlin.reflect.jvm.internal.impl.types.AbstractTypeConstructor$supertypes$1.invoke(AbstractTypeConstructor.kt:83)
    at kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedLazyValue.invoke(LockBasedStorageManager.java:408)
    at kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedLazyValueWithPostCompute.invoke(LockBasedStorageManager.java:481)
    at kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedNotNullLazyValueWithPostCompute.invoke(LockBasedStorageManager.java:512)
    at kotlin.reflect.jvm.internal.impl.types.AbstractTypeConstructor.getSupertypes(AbstractTypeConstructor.kt:30)
    at kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope.getNonDeclaredVariableNames(DeserializedClassDescriptor.kt:309)
    at kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation$variableNames$2.invoke(DeserializedMemberScope.kt:262)
    at kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation$variableNames$2.invoke(DeserializedMemberScope.kt:261)
    at kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedLazyValue.invoke(LockBasedStorageManager.java:408)
    at kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedNotNullLazyValue.invoke(LockBasedStorageManager.java:527)
    at kotlin.reflect.jvm.internal.impl.storage.StorageKt.getValue(storage.kt:42)
    at kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation.getVariableNames(DeserializedMemberScope.kt:261)
    at kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation.addFunctionsAndPropertiesTo(DeserializedMemberScope.kt:349)
    at kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope.computeDescriptors(DeserializedMemberScope.kt:115)
    at kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope$allDescriptors$1.invoke(DeserializedClassDescriptor.kt:230)
    at kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope$allDescriptors$1.invoke(DeserializedClassDescriptor.kt:229)
    at kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedLazyValue.invoke(LockBasedStorageManager.java:408)
    at kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedNotNullLazyValue.invoke(LockBasedStorageManager.java:527)
    at kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope.getContributedDescriptors(DeserializedClassDescriptor.kt:240)
    at kotlin.reflect.jvm.internal.impl.resolve.scopes.ResolutionScope$DefaultImpls.getContributedDescriptors$default(ResolutionScope.kt:50)
    at kotlin.reflect.jvm.internal.KDeclarationContainerImpl.getMembers(KDeclarationContainerImpl.kt:56)
    at kotlin.reflect.jvm.internal.KClassImpl$Data$declaredNonStaticMembers$2.invoke(KClassImpl.kt:162)
    at kotlin.reflect.jvm.internal.KClassImpl$Data$declaredNonStaticMembers$2.invoke(KClassImpl.kt:47)
    at kotlin.reflect.jvm.internal.ReflectProperties$LazySoftVal.invoke(ReflectProperties.java:92)
    at kotlin.reflect.jvm.internal.ReflectProperties$Val.getValue(ReflectProperties.java:31)
    at kotlin.reflect.jvm.internal.KClassImpl$Data.getDeclaredNonStaticMembers(KClassImpl.kt)
    at kotlin.reflect.jvm.internal.KClassImpl$Data$allNonStaticMembers$2.invoke(KClassImpl.kt:171)
    at kotlin.reflect.jvm.internal.KClassImpl$Data$allNonStaticMembers$2.invoke(KClassImpl.kt:47)
    at kotlin.reflect.jvm.internal.ReflectProperties$LazySoftVal.invoke(ReflectProperties.java:92)
    at kotlin.reflect.jvm.internal.ReflectProperties$Val.getValue(ReflectProperties.java:31)
    at kotlin.reflect.jvm.internal.KClassImpl$Data.getAllNonStaticMembers(KClassImpl.kt)
    at kotlin.reflect.full.KClasses.getMemberProperties(KClasses.kt:149)
    at auto_resources.d.b(Unknown Source)
    at auto_resources.d.a(Unknown Source)
    at main.u.a(Unknown Source)
    at aN.c(Unknown Source)
    at aN.a(Unknown Source)
    at main.O.<init>(Unknown Source)
    at main.w.c(Unknown Source)
    at main.w.a(Unknown Source)
    at qU.b(Unknown Source)
    at main.NebRaxApp.a(Unknown Source)
    at main.NebRaxApp.main(Unknown Source)

While neither DemoPointLightBehavior or GameBehavior (it's parent class) appear in the ProGuard log, presumably the reported class names should be obfuscated, but are not. (I checked the jar, and both .class file names are indeed obfuscated.) Perhaps the obfuscator is totally missing the metadata? GameBehavior is the root class and DemoPointLightBehavior is its direct descendent.

marshall86 commented 3 weeks ago

i'm facing a similar issue using dexguard, does anybody sort this out? thanks

mrjameshamilton commented 3 weeks ago

Hi @marshall86 ! For DexGuard support, please reach out via the usual customer support channels to get help with your issue.