Problem of running soot on latest version Spark

[leafeonia]

By downloading latest version in /app and modifying code in /acai, Hadoop is successfully ported to the latest version. However, the same method fails when porting Spark from spark-2.4.5-bin-hadoop2.7 to latest version ( I tried both spark-3.0.1-bin-hadoop2.7 and spark-3.0.1-bin-hadoop3.2 ). Running ccc and it comes to an error:

Exception in thread "Thread-2" java.lang.RuntimeException: Failed to apply jb to <org.apache.spark.streaming.scheduler.RateController: void $anonfun$computeAndPublish$1(org.apache.spark.streaming.scheduler.RateController,long,long,long,long)>
    at soot.asm.AsmMethodSource.getBody(AsmMethodSource.java:2191)
    at soot.SootMethod.retrieveActiveBody(SootMethod.java:402)
    at soot.PackManager$1.run(PackManager.java:1279)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: This operation requires resolving level HIERARCHY but scala.runtime.java8.JFunction1$mcVD$sp is at resolving level DANGLING
If you are extending Soot, try to add the following call before calling soot.Main.main(..):
Scene.v().addBasicClass(scala.runtime.java8.JFunction1$mcVD$sp,HIERARCHY);
Otherwise, try whole-program mode (-w).
    at soot.SootClass.checkLevelIgnoreResolving(SootClass.java:198)
    at soot.SootClass.checkLevel(SootClass.java:180)
    at soot.FastHierarchy.canStoreClass(FastHierarchy.java:346)
    at soot.FastHierarchy.canStoreType(FastHierarchy.java:267)
    at soot.jimple.toolkits.typing.fast.BytecodeHierarchy.ancestor_(BytecodeHierarchy.java:228)
    at soot.jimple.toolkits.typing.fast.BytecodeHierarchy.ancestor(BytecodeHierarchy.java:296)
    at soot.jimple.toolkits.typing.fast.TypeResolver$CastInsertionUseVisitor.visit(TypeResolver.java:213)
    at soot.jimple.toolkits.typing.fast.UseChecker.handleInvokeExpr(UseChecker.java:140)
    at soot.jimple.toolkits.typing.fast.UseChecker.caseInvokeStmt(UseChecker.java:182)
    at soot.jimple.internal.JInvokeStmt.apply(JInvokeStmt.java:91)
    at soot.jimple.toolkits.typing.fast.UseChecker.check(UseChecker.java:127)
    at soot.jimple.toolkits.typing.fast.TypeResolver.insertCasts(TypeResolver.java:443)
    at soot.jimple.toolkits.typing.fast.TypeResolver.minCasts(TypeResolver.java:467)
    at soot.jimple.toolkits.typing.fast.TypeResolver.inferTypes(TypeResolver.java:156)
    at soot.jimple.toolkits.typing.TypeAssigner.internalTransform(TypeAssigner.java:121)
    at soot.BodyTransformer.transform(BodyTransformer.java:51)
    at soot.Transform.apply(Transform.java:120)
    at soot.JimpleBodyPack.applyPhaseOptions(JimpleBodyPack.java:73)
    at soot.JimpleBodyPack.internalApply(JimpleBodyPack.java:105)
    at soot.Pack.apply(Pack.java:125)
    at soot.asm.AsmMethodSource.getBody(AsmMethodSource.java:2189)
    ... 5 more

I tried to fix this problem by adapting 2 suggestions listed in the log above:

If you are extending Soot, try to add the following call before calling soot.Main.main(..):
Scene.v().addBasicClass(scala.runtime.java8.JFunction1$mcVD$sp,HIERARCHY);

Otherwise, try whole-program mode (-w).

So I modify the code in acai/src/main/java/acai/utility/Util.java, where parameters are set and soot.Main.main() gets called ( lines with '+' at start are added by me):

public class Util {

    public static Set<String> getAppClasses(List<String> appPaths) {
        soot.G.reset();

        String classPath = String.join(":", appPaths);
        String[] initArgs = {
+               "-w",                       

                // Input Options
                "-cp", classPath,
                "-pp",
                "-allow-phantom-refs",
                "-no-bodies-for-excluded",

                // Output Options
                "-f", "n",
        };

        String[] sootArgs = new String[initArgs.length + 2 * appPaths.size()];
        for (int i = 0; i < initArgs.length; i++) {
            sootArgs[i] = initArgs[i];
        }
        for (int i = 0; i < appPaths.size(); i++) {
            sootArgs[initArgs.length + 2*i] = "-process-dir";
            sootArgs[initArgs.length + 2*i + 1] = appPaths.get(i);
        }

+       Scene.v().addBasicClass("scala.runtime.java8.JFunction1$mcVD$sp",SootClass.HIERARCHY);

        soot.Main.main(sootArgs);

        Set<String> appClasses = new HashSet<>();
        for (SootClass sc : Scene.v().getApplicationClasses()) {
            appClasses.add(sc.toString());
        }

        return appClasses;
    }

    public static List<String> getEntryPoints(List<String> srcPaths) {
        soot.G.reset();

        String classPath = String.join(":", srcPaths);
        String[] initArgs = {
                // General Options
                "-w",

                // Input Options
                "-cp", classPath,
                "-pp",
                "-allow-phantom-refs",
                "-no-bodies-for-excluded",

                // Output Options
                "-f", "n",

                // Phase Options
                "-p", "cg", "off",
        };

        String[] sootArgs = new String[initArgs.length + 2 * srcPaths.size()];
        for (int i = 0; i < initArgs.length; i++) {
            sootArgs[i] = initArgs[i];
        }
        for (int i = 0; i < srcPaths.size(); i++) {
            sootArgs[initArgs.length + 2*i] = "-process-dir";
            sootArgs[initArgs.length + 2*i + 1] = srcPaths.get(i);
        }

+       Scene.v().addBasicClass("scala.runtime.java8.JFunction1$mcVD$sp",SootClass.HIERARCHY);

        soot.Main.main(sootArgs);

        // Extract entry points for all-reachable call graph generation
        List<SootMethod> entryPoints = new ArrayList<>();
        entryPoints.addAll(EntryPoints.v().methodsOfApplicationClasses());
        List<String> entryPointsStr = entryPoints
                .stream()
                .map(m -> m.toString())
                .collect(Collectors.toList());

        return entryPointsStr;
    }

}

However, the problem still exists. From the log information that's the only methods I can think of. I also searched for solution on Internet but did not find much information available. If you meet the same problem or have some suggstions, please post on the comment.

[X-czh]

I reproduced the problem and have no idea how to solve this. The only thing we can do is to submit an issue to Soot and stick to the old version for now.

[leafeonia]

I reproduced the problem and have no idea how to solve this. The only thing we can do is to submit an issue to Soot and stick to the old version for now.

I have submitted an issue here.

[tianyin]

Now you know why I say it's not that easy to port :)

I agree that the error is in the Soot side, and it's not easy to fix unless you know Soot really well.

It's totally fine to keep use the old version, so you can focus on the more important tasks.

[leafeonia]

Now you know why I say it's not that easy to port :)

I agree that the error is in the Soot side, and it's not easy to fix unless you know Soot really well.

It's totally fine to keep use the old version, so you can focus on the more important tasks.

Soot did a good job in collaborating with you by offering an example of problem and teach me a good lesson lol 😅