search

soot-oss / soot

Soot - A Java optimization framework
GNU Lesser General Public License v2.1
2.87k stars 706 forks source link

ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed #1067

Open zhou-shulin opened 5 years ago

zhou-shulin commented 5 years ago

I am using soot to do some static analysis on Java program. I wrote an analysis transformer in 'wjtp" phase that only read information from jimple code, nothing is changed. But when I ran the transformer on HBase-2.1.1 (including hbase-common-2.1.1.jar, hbase-server-2.1.1.jar and hbase-client-2.1.1.jar), here comes a problem as follows (The detailed stack information is in attachment-1.txt):

[Thread-6] ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: Trying to create interface invoke expression for non-interface type: org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Use JVirtualInvokeExpr or JSpecialInvokeExpr instead!
java.lang.RuntimeException: Trying to create interface invoke expression for non-interface type: org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Use JVirtualInvokeExpr or JSpecialInvokeExpr instead!
    at soot.jimple.internal.JInterfaceInvokeExpr.<init>(JInterfaceInvokeExpr.java:51)
    at soot.jimple.internal.JInterfaceInvokeExpr.clone(JInterfaceInvokeExpr.java:67)
    at soot.jimple.Jimple.cloneIfNecessary(Jimple.java:144)
    at soot.jimple.internal.JAssignStmt.clone(JAssignStmt.java:221)
    at soot.Body.importBodyContentsFrom(Body.java:165)
    at soot.jimple.JimpleBody.clone(JimpleBody.java:87)
    at soot.PackManager.convertJimpleBodyToBaf(PackManager.java:1030)
    at soot.PackManager.runBodyPacks(PackManager.java:991)
    at soot.PackManager.access$000(PackManager.java:112)
    at soot.PackManager$1.run(PackManager.java:620)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

The source code of this org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory is in "hbase/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceFactory.java" as follows, it is clear that this MetricsRegionServerSourceFactory is a interface. Also I have used the options "-allow-phantom-refs" to avoid this kind of unknown class to be resolve incorrectly.

package org.apache.hadoop.hbase.regionserver;

import org.apache.hadoop.hbase.io.MetricsIOSource;
import org.apache.hadoop.hbase.io.MetricsIOWrapper;
import org.apache.yetus.audience.InterfaceAudience;

/**
 * Interface of a factory to create Metrics Sources used inside of regionservers.
 */
@InterfaceAudience.Private
public interface MetricsRegionServerSourceFactory {
   ... ...
}

Then I added the option "-ignore-resolution-errors" in my transformer, but there came another exception as follows:

Exception in thread "main" java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:907)
    at java.util.ArrayList$Itr.next(ArrayList.java:857)
    at soot.jimple.toolkits.annotation.LineNumberAdder.internalTransform(LineNumberAdder.java:42)
    at soot.PackManager.runPacksNormally(PackManager.java:451)
    at soot.PackManager.runPacks(PackManager.java:384)
        at Main.main(Main.java:137)

Next, I added another jar file (hbase-hadoop-compat-2.1.1.jar) to analyse, and no problem came yet.

However, when I added all the jar files in hbase-2.1.1-bin.tar.gz from official websites to analyse. It came another problem as follows:

Exception in thread "main" java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(Unknown Source)
    at java.util.ArrayList$Itr.next(Unknown Source)
    at indi.abc.MyTransformer.analyzeSootClass(MatcherTransformer.java:125)
    at indi.abc.MyTransformer.internalTransform(MatcherTransformer.java:70)
    at soot.SceneTransformer.transform(SceneTransformer.java:39)
    at soot.Transform.apply(Transform.java:90)
    at soot.ScenePack.internalApply(ScenePack.java:40)
    at soot.Pack.apply(Pack.java:118)
    at soot.PackManager.runWholeProgramPacks(PackManager.java:565)
    at soot.PackManager.runPacksNormally(PackManager.java:455)
    at soot.PackManager.runPacks(PackManager.java:384)
        at Main.main(Main.java:137)

Then I changed the ArrayList as CopyOnWriteArrayList to avoid ConcurrentModificationException, it still has some weird problems as follows:

[Thread-7] ERROR heros.solver.CountingThreadPoolExecutor - Worker thread execution failed: null
java.lang.NullPointerException
    at soot.jimple.validation.FieldRefValidator.validate(FieldRefValidator.java:52)
    at soot.jimple.JimpleBody.validate(JimpleBody.java:114)
    at soot.jimple.JimpleBody.validate(JimpleBody.java:97)
    at soot.baf.BafBody.<init>(BafBody.java:88)
    at soot.baf.Baf.newBody(Baf.java:560)
    at soot.PackManager.convertJimpleBodyToBaf(PackManager.java:1036)
    at soot.PackManager.runBodyPacks(PackManager.java:991)
    at soot.PackManager.access$000(PackManager.java:112)
    at soot.PackManager$1.run(PackManager.java:620)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Exception in thread "Thread-7" java.lang.NullPointerException
    at soot.jimple.validation.FieldRefValidator.validate(FieldRefValidator.java:52)
    at soot.jimple.JimpleBody.validate(JimpleBody.java:114)
    at soot.jimple.JimpleBody.validate(JimpleBody.java:97)
    at soot.baf.BafBody.<init>(BafBody.java:88)
    at soot.baf.Baf.newBody(Baf.java:560)
    at soot.PackManager.convertJimpleBodyToBaf(PackManager.java:1036)
    at soot.PackManager.runBodyPacks(PackManager.java:991)
    at soot.PackManager.access$000(PackManager.java:112)
    at soot.PackManager$1.run(PackManager.java:620)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Exception in thread "main" java.lang.NullPointerException
    at soot.jimple.validation.FieldRefValidator.validate(FieldRefValidator.java:52)
    at soot.jimple.JimpleBody.validate(JimpleBody.java:114)
    at soot.jimple.JimpleBody.validate(JimpleBody.java:97)
    at soot.baf.BafBody.<init>(BafBody.java:88)
    at soot.baf.Baf.newBody(Baf.java:560)
    at soot.PackManager.convertJimpleBodyToBaf(PackManager.java:1036)
    at soot.PackManager.runBodyPacks(PackManager.java:991)
    at soot.PackManager.access$000(PackManager.java:112)
    at soot.PackManager$1.run(PackManager.java:620)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

What is the problem? Why this single thread analysis without any deletion on elements could induce ConcurrentModificationException ? And what is the root cause of the last exception?

Hope for guidance sincerely!

mbenz89 commented 5 years ago

Could you post your analysis code?

The first exception seems to be caused by incorrect type inference conducted by Soot. It might be related to #999. The last one might be related to this problem when ignoring resolution errors, but this is just a guess.

Could you further investigate the cause of these issues? I assume finding the cause of the ConcurrentModificationException and solve it might be the easiest of those three...