soot-oss / soot

Soot - A Java optimization framework
GNU Lesser General Public License v2.1
2.89k stars 710 forks source link

Constant value for field mismatch between code (40) and constant table (ConstantValue: 0) #969

Open KevinLeigh opened 6 years ago

KevinLeigh commented 6 years ago

Hey there, I am using Soot and am getting the following issue below:

E AndroidRuntime: java.lang.VerifyError: Verifier rejected class com.facebook.nobreak.CatchMeIfYouCan: void com.facebook.nobreak.CatchMeIfYouCan.killThisProcess(): [0xFFFFFFFF] branch offset of zero not allowed at0x1a (declaration of 'com.facebook.nobreak.CatchMeIfYouCan' appears in /data/app/com.facebook.katana-1/base.apk)
E AndroidRuntime:   at com.facebook.nobreak.CatchMeIfYouCan.init()
E AndroidRuntime:   at com.facebook.katana.app.FacebookApplication.O()
E AndroidRuntime:   at X.000.G()
E AndroidRuntime:   at X.001.attachBaseContext()
E AndroidRuntime:   at android.app.Application.attach(Application.java:189)
E AndroidRuntime:   at android.app.Instrumentation.newApplication(Instrumentation.java:1008)
E AndroidRuntime:   at android.app.Instrumentation.newApplication(Instrumentation.java:992)
E AndroidRuntime:   at android.app.LoadedApk.makeApplication(LoadedApk.java:796)
E AndroidRuntime:   at android.app.ActivityThread.handleBindApplication(ActivityThread.java:5377)
E AndroidRuntime:   at android.app.ActivityThread.-wrap2(ActivityThread.java)
E AndroidRuntime:   at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1545)
E AndroidRuntime:   at android.os.Handler.dispatchMessage(Handler.java:102)
E AndroidRuntime:   at android.os.Looper.loop(Looper.java:154)
E AndroidRuntime:   at android.app.ActivityThread.main(ActivityThread.java:6119)
E AndroidRuntime:   at java.lang.reflect.Method.invoke(Native Method)
E AndroidRuntime:   at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886)
E AndroidRuntime:   at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776)
W ActivityManager:   Force finishing activity com.facebook.katana/.LoginActivity

And believe it relates to this warning from Soot compilation process below:

WARNING: Constant value for field '<com.facebook.nobreak.CatchMeIfYouCan: int NR_CRASH_LOG_RECORDS>' mismatch between code (40) and constant table (ConstantValue: 0)

It could be that actual byte code instruction addresses are not matching when being disassembled with dexlib2

KevinLeigh commented 6 years ago

Hey Guys, any update on the above issue?

mbenz89 commented 6 years ago

Hi. I'm not even sure what you are doing. Are you analyzing and writing back an APK that afterward breaks? Maybe you can track down the error yourself first. At a first step, it might make sense to update the dexlib2 version to 2.2.4 and check if the error remains.

KevinLeigh commented 6 years ago

Hey @mbenz89.

Thank you for the response, currently I am writing back to an APK using Soot, I will try resolve the issue by cloning this repo and updating dexlib2 from 2.2.2 -> `2.2.4, the will create a new jar to test against. Is there any gotchas for creating a jar with this repo?

mbenz89 commented 6 years ago

By executing mvn package -DskipTests, Soot will be built and a Jar containing all dependencies will be placed in the targets directory. For debugging purposes, it might be easier to include Soot as a maven project into your IDE and configure your Project to give the Soot project priority over your current Soot dependency (depending on your configuration, this might happen automatically).

KevinLeigh commented 6 years ago

@mbenz89 Perfect thank you. The information provided will definitely aid in building the Soot project for testing dexlib2 library update. Will add Soot as project dependency as mentioned it will be more useful for debugging purposes.

KevinLeigh commented 6 years ago

So I updated the dexlib2 library in Soot to 2.2.4 and tried again with project the writes back to Apk. Unfortunately the same issue still occurs. Will investigate further into dexlib2 and why the output would result in a byte code mismatch when Apk is disassembled.

mbenz89 commented 6 years ago

Could you figure out which type the right operand of the assignment has (stated as code 40 in the error message)?

KevinLeigh commented 6 years ago

@mbenz89 Hey there hope this debug screenshot helps, please see below:

screen shot 2018-06-22 at 14 09 41
mbenz89 commented 6 years ago

Ok, so for some reason, the tag assigned to the field has a value of 0 while the right operand of the assignment has a value of 40. So the value in the constant table 0 is different to the one that was used in the assignment in the static analyzer.

Actually, if I think about it, that should be totally ok. I think it only means that the constant table does not hold the correct initializer value and the correct value (40) is assigned during class initialization. If they would match (so constant table and assigned value are equal), the assignment statement could have been removed due to being redundant.

The error you are observing [0xFFFFFFFF] branch offset of zero not allowed at0x1a might very well be completely unrelated to the warning about the constants value mismatch.

It seems more as a branching target is wrongly calculated. Sorry but I guess you have to track that down a bit more.

KevinLeigh commented 6 years ago

@mbenz89 Thank you for the help, what you saying does sound correct and I think its further down as well, because if I re-write any method in that class the same crash error occurs but points to another part of the same class(CatchMeIfYouCan). It really seems like there is a issue with the alignment of bytes when dexlib2 disassembles that specific class. The issue causes the Apk to crash when run.

KevinLeigh commented 6 years ago

Hey there @mbenz89,

I have look further into the logcat for instrumented Apk, there is a lot of verification errors thrown by dex2oat. It could be because of trap handlers during transformation phase of soot. Is there maybe a way to disable trap handlers or transformation of a specific class to keep the bytecode identical? Please advise

W/dex2oat: Unexpected CPU variant for X86 using defaults: x86
2018-07-20 17:44:08.740 8661-8661/? W/dex2oat: Mismatch between dex2oat instruction set features (ISA: X86 Feature string: smp,-ssse3,-sse4.1,-sse4.2,-avx,-avx2,-lock_add,-popcnt) and those of dex2oat executable (ISA: X86 Feature string: smp,ssse3,-sse4.1,-sse4.2,-avx,-avx2,-lock_add,-popcnt) for the command line:
2018-07-20 17:44:08.740 8661-8661/? W/dex2oat: /system/bin/dex2oat --zip-fd=6 --zip-location=base.apk --oat-fd=7 --oat-location=/data/app/vmdl1010003394.tmp/oat/x86/base.odex --instruction-set=x86 --instruction-set-variant=x86 --instruction-set-features=default --runtime-arg -Xms64m --runtime-arg -Xmx512m --compiler-filter=interpret-only --swap-fd=9 --runtime-arg -classpath --runtime-arg /system/framework/com.google.android.maps.jar
2018-07-20 17:44:08.740 8661-8661/? I/dex2oat: /system/bin/dex2oat --compiler-filter=interpret-only
2018-07-20 17:44:08.869 8661-8665/? W/dex2oat: Verification error in void X.006.F(java.lang.String, java.lang.String, java.lang.String, int, android.os.StrictMode$ThreadPolicy)
2018-07-20 17:44:08.869 8661-8665/? W/dex2oat: void X.006.F(java.lang.String, java.lang.String, java.lang.String, int, android.os.StrictMode$ThreadPolicy) failed to verify: void X.006.F(java.lang.String, java.lang.String, java.lang.String, int, android.os.StrictMode$ThreadPolicy): [0x87] copy1 v26<-v27 type=Undefined cat=3
2018-07-20 17:44:08.883 8661-8663/? W/dex2oat: Verification error in void X.00j.uncaughtException(java.lang.Thread, java.lang.Throwable)
2018-07-20 17:44:08.883 8661-8663/? W/dex2oat: void X.00j.uncaughtException(java.lang.Thread, java.lang.Throwable): [0xFFFFFFFF] branch offset of zero not allowed at0xe6
2018-07-20 17:44:08.895 8661-8661/? W/dex2oat: Verification error in void X.00p.A(X.079)
2018-07-20 17:44:08.895 8661-8661/? W/dex2oat: void X.00p.A(X.079) failed to verify: void X.00p.A(X.079): [0x75] thrown value of non-reference type Undefined
2018-07-20 17:44:08.914 8661-8663/? W/dex2oat: Verification error in void X.00o.run()
2018-07-20 17:44:08.914 8661-8663/? W/dex2oat: void X.00o.run() failed to verify: void X.00o.run(): [0x0] copy1 v4<-v5 type=Undefined cat=3
2018-07-20 17:44:08.914 8661-8665/? W/dex2oat: Verification error in int X.01a.F(java.lang.String, int, java.io.File, android.os.StrictMode$ThreadPolicy)
2018-07-20 17:44:08.914 8661-8665/? W/dex2oat: int X.01a.F(java.lang.String, int, java.io.File, android.os.StrictMode$ThreadPolicy) failed to verify: int X.01a.F(java.lang.String, int, java.io.File, android.os.StrictMode$ThreadPolicy): [0x613] copy1 v281<-v282 type=Undefined cat=3
2018-07-20 17:44:08.915 8661-8661/? W/dex2oat: Rejecting class X.01e that attempts to sub-type erroneous class X.01a in base.apk
2018-07-20 17:44:08.916 8661-8661/? W/dex2oat: Rejecting class X.01i that attempts to sub-type erroneous class X.01e in base.apk
2018-07-20 17:44:08.921 8661-8664/? W/dex2oat: Rejecting class X.01d that attempts to sub-type erroneous class X.01e in base.apk
2018-07-20 17:44:08.922 8661-8665/? W/dex2oat: Rejecting class X.01g that attempts to sub-type erroneous class X.01d in base.apk
2018-07-20 17:44:08.922 8661-8663/? W/dex2oat: Rejecting class X.01c that attempts to sub-type erroneous class X.01d in base.apk
2018-07-20 17:44:09.013 8661-8664/? W/dex2oat: Class X.07v failed lock verification and will run slower.
2018-07-20 17:44:09.013 8661-8664/? W/dex2oat: Common causes for lock verification issues are non-optimized dex code
2018-07-20 17:44:09.013 8661-8664/? W/dex2oat: and incorrect proguard optimizations.
2018-07-20 17:44:09.023 8661-8661/? W/dex2oat: Class X.091 failed lock verification and will run slower.
2018-07-20 17:44:09.028 8661-8663/? W/dex2oat: Verification error in void X.07R.V(java.lang.String)
2018-07-20 17:44:09.028 8661-8663/? W/dex2oat: void X.07R.V(java.lang.String) failed to verify: void X.07R.V(java.lang.String): [0xD7] copy1 v1<-v55 type=Undefined cat=3
2018-07-20 17:44:09.029 8661-8663/? W/dex2oat: Verification error in void X.07R.Z(java.util.List)
2018-07-20 17:44:09.029 8661-8663/? W/dex2oat: void X.07R.Z(java.util.List) failed to verify: void X.07R.Z(java.util.List): [0x99] register v13 has type Undefined but expected Reference: java.lang.Throwable
2018-07-20 17:44:09.029 8661-8663/? W/dex2oat: Verification error in void X.07R.b(java.util.List)
2018-07-20 17:44:09.029 8661-8663/? W/dex2oat: void X.07R.b(java.util.List) failed to verify: void X.07R.b(java.util.List): [0x99] register v13 has type Undefined but expected Reference: java.lang.Throwable
2018-07-20 17:44:09.030 8661-8665/? W/dex2oat: Verification error in void X.09u.run()
2018-07-20 17:44:09.030 8661-8665/? W/dex2oat: void X.09u.run() failed to verify: void X.09u.run(): [0x3B9] copy1 v183<-v184 type=Undefined cat=3
2018-07-20 17:44:09.040 8661-8661/? W/dex2oat: Class X.0AR failed lock verification and will run slower.
2018-07-20 17:44:09.042 8661-8664/? W/dex2oat: Verification error in void X.0Ac.G()
2018-07-20 17:44:09.042 8661-8664/? W/dex2oat: void X.0Ac.G() failed to verify: void X.0Ac.G(): [0x103] copy1 v2<-v32 type=Undefined cat=3
2018-07-20 17:44:09.043 8661-8665/? W/dex2oat: Verification error in void X.0At.IbD(long)
2018-07-20 17:44:09.043 8661-8665/? W/dex2oat: void X.0At.IbD(long) failed to verify: void X.0At.IbD(long): [0x14] copy1 v5<-v6 type=Undefined cat=3
2018-07-20 17:44:09.043 8661-8664/? E/dex2oat: Verification failed on class X.0Ac because: Verifier rejected class X.0Ac: void X.0Ac.G() failed to verify: void X.0Ac.G(): [0x103] copy1 v2<-v32 type=Undefined cat=3
2018-07-20 17:44:09.053 8661-8661/? W/dex2oat: Verification error in void X.0C9.run()
2018-07-20 17:44:09.053 8661-8661/? W/dex2oat: void X.0C9.run() failed to verify: void X.0C9.run(): [0x0] copy1 v11<-v12 type=Undefined cat=3
2018-07-20 17:44:09.054 8661-8665/? W/dex2oat: Verification error in java.lang.String X.0C4.pCB()
2018-07-20 17:44:09.054 8661-8665/? W/dex2oat: java.lang.String X.0C4.pCB(): [0xFFFFFFFF] bad exception entry: startAddr=19 endAddr=19 (size=25)
2018-07-20 17:44:09.055 8661-8665/? W/dex2oat: Class X.0C4 failed lock verification and will run slower.
2018-07-20 17:44:09.058 8661-8664/? W/dex2oat: Class X.0CI failed lock verification and will run slower.
2018-07-20 17:44:09.076 8661-8661/? W/dex2oat: Rejecting class X.0H0 that attempts to sub-type erroneous class X.01e in base.apk
2018-07-20 17:44:09.081 8661-8664/? W/dex2oat: Verification error in com.facebook.acra.Spool$FileBeingConsumed com.facebook.acra.Spool$Snapshot.tryOpenFileForConsumption(com.facebook.acra.Spool$Descriptor)
2018-07-20 17:44:09.081 8661-8664/? W/dex2oat: com.facebook.acra.Spool$FileBeingConsumed com.facebook.acra.Spool$Snapshot.tryOpenFileForConsumption(com.facebook.acra.Spool$Descriptor) failed to verify: com.facebook.acra.Spool$FileBeingConsumed com.facebook.acra.Spool$Snapshot.tryOpenFileForConsumption(com.facebook.acra.Spool$Descriptor): [0xAA] copy1 v37<-v38 type=Undefined cat=3
2018-07-20 17:44:09.082 8661-8661/? W/dex2oat: Verification error in com.facebook.acra.ErrorReporter$ReportsSenderWorker com.facebook.acra.ErrorReporter.handleExceptionInternal(java.lang.Throwable, java.util.Map, java.lang.String, int)
2018-07-20 17:44:09.082 8661-8661/? W/dex2oat: com.facebook.acra.ErrorReporter$ReportsSenderWorker com.facebook.acra.ErrorReporter.handleExceptionInternal(java.lang.Throwable, java.util.Map, java.lang.String, int) failed to verify: com.facebook.acra.ErrorReporter$ReportsSenderWorker com.facebook.acra.ErrorReporter.handleExceptionInternal(java.lang.Throwable, java.util.Map, java.lang.String, int): [0xEB] thrown value of non-reference type Undefined
2018-07-20 17:44:09.086 8661-8661/? W/dex2oat: Verification error in java.lang.Throwable com.facebook.acra.ErrorReporter.translateException(java.lang.Throwable, java.util.Map)
2018-07-20 17:44:09.086 8661-8661/? W/dex2oat: java.lang.Throwable com.facebook.acra.ErrorReporter.translateException(java.lang.Throwable, java.util.Map) failed to verify: java.lang.Throwable com.facebook.acra.ErrorReporter.translateException(java.lang.Throwable, java.util.Map): [0xF] copy1 v9<-v10 type=Undefined cat=3
2018-07-20 17:44:09.098 8661-8663/? W/dex2oat: Verification error in void com.facebook.breakpad.BreakpadManager.start(android.content.Context, int, int)
2018-07-20 17:44:09.098 8661-8663/? W/dex2oat: void com.facebook.breakpad.BreakpadManager.start(android.content.Context, int, int): [0xFFFFFFFF] bad exception entry: startAddr=213 endAddr=213 (size=285)
2018-07-20 17:44:09.099 8661-8665/? W/dex2oat: Verification error in void com.facebook.common.dextricks.ClassFailureStapler.onClassInitFailure(java.lang.Class, java.lang.Throwable)
2018-07-20 17:44:09.099 8661-8665/? W/dex2oat: void com.facebook.common.dextricks.ClassFailureStapler.onClassInitFailure(java.lang.Class, java.lang.Throwable) failed to verify: void com.facebook.common.dextricks.ClassFailureStapler.onClassInitFailure(java.lang.Class, java.lang.Throwable): [0x2C] thrown value of non-reference type Undefined
2018-07-20 17:44:09.099 8661-8665/? W/dex2oat: Verification error in java.lang.String com.facebook.common.dextricks.DexErrorRecoveryInfo.toString()
2018-07-20 17:44:09.099 8661-8665/? W/dex2oat: java.lang.String com.facebook.common.dextricks.DexErrorRecoveryInfo.toString() failed to verify: java.lang.String com.facebook.common.dextricks.DexErrorRecoveryInfo.toString(): [0xC] copy1 v5<-v6 type=Undefined cat=3
2018-07-20 17:44:09.100 8661-8664/? W/dex2oat: Verification error in java.lang.String com.facebook.common.dextricks.DexDiagnostics.dumpLogcatData()
2018-07-20 17:44:09.100 8661-8664/? W/dex2oat: java.lang.String com.facebook.common.dextricks.DexDiagnostics.dumpLogcatData() failed to verify: java.lang.String com.facebook.common.dextricks.DexDiagnostics.dumpLogcatData(): [0x1F0] copy1 v0<-v82 type=Undefined cat=3
2018-07-20 17:44:09.105 8661-8664/? W/dex2oat: Verification error in void com.facebook.common.dextricks.DexStore.installCrossDexHooks()
2018-07-20 17:44:09.105 8661-8664/? W/dex2oat: void com.facebook.common.dextricks.DexStore.installCrossDexHooks() failed to verify: void com.facebook.common.dextricks.DexStore.installCrossDexHooks(): [0x3] copy1 v1<-v2 type=Undefined cat=3
2018-07-20 17:44:09.113 8661-8663/? W/dex2oat: Verification error in com.facebook.common.dextricks.ReentrantLockFile$Lock com.facebook.common.dextricks.ReentrantLockFile.acquireInterruptubly(int)
2018-07-20 17:44:09.113 8661-8663/? W/dex2oat: com.facebook.common.dextricks.ReentrantLockFile$Lock com.facebook.common.dextricks.ReentrantLockFile.acquireInterruptubly(int) failed to verify: com.facebook.common.dextricks.ReentrantLockFile$Lock com.facebook.common.dextricks.ReentrantLockFile.acquireInterruptubly(int): [0x15] copy1 v13<-v14 type=Undefined cat=3
2018-07-20 17:44:09.120 8661-8661/? W/dex2oat: Verification error in void com.facebook.nobreak.CatchMeIfYouCan.killThisProcess()
2018-07-20 17:44:09.120 8661-8661/? W/dex2oat: void com.facebook.nobreak.CatchMeIfYouCan.killThisProcess(): [0xFFFFFFFF] branch offset of zero not allowed at0xc
2018-07-20 17:44:09.460 8661-8661/? I/dex2oat: dex2oat took 720.100ms (threads: 4) arena alloc=2012KB (2061104B) java alloc=1291KB (1322560B) native alloc=2MB (2130008B) free=3MB (3637160B)
KevinLeigh commented 6 years ago

Hey there, any update on the above comment?

mbenz89 commented 6 years ago

You could try the Trap Tightener (jb.tt) phase. Not sure it can help here, though.

I am not aware of the possibility to deactivate trap handling. For testing purposes, you might have the opportunity to exclude the stated problematic class so that it is not processed by Soot and then manually instrument in back into the generated dex after writing out. This should leave the class' bytecode as is.

KevinLeigh commented 6 years ago

@mbenz89 Thank you for the response, much appreciated. Will definitely try out jb.tt. Do you have an example of how I can set jb.tt before calling soot.Main?

Do you also have a sample of how I can manually get class and instrument it back into the dex after soot has processed ?

mbenz89 commented 6 years ago
  1. -p jb.tt enabled:true as option to soot.Main
  2. I've never done it myself but I guess there are several tools you can use. What comes into my mind, is using apktool to extract the smali class file of your class from the original apk and then repack your Soot generated apk with this smali class added.

So:

KevinLeigh commented 6 years ago

@mbenz89 Thank you so much! Will definitely give it a try, and post and update shortly after to let you know if it worked.

KevinLeigh commented 6 years ago

@mbenz89 Hey there, I tried both suggestions with no luck, the most promising one was using ApkTool, however I get a recompile error which is an open issue on ApkTool repo here -> https://github.com/iBotPeaches/Apktool/issues/1719. Thank you for your help, much appreciated. I will have to await to see if issue gets resolved before I can try again.