NullPointerException while building spark callgraph.

[giltho]

Hello,

I loaded packages (that are unfortunately not open-source) and built the callgraph using spark & VTA, and obtained this error:

Exception in thread "main" java.lang.NullPointerException
    at soot.jimple.spark.builder.MethodNodeFactory.caseStaticFieldRef(MethodNodeFactory.java:404)
    at soot.jimple.StaticFieldRef.apply(StaticFieldRef.java:83)
    at soot.jimple.spark.builder.MethodNodeFactory$1.caseAssignStmt(MethodNodeFactory.java:162)
    at soot.jimple.internal.JAssignStmt.apply(JAssignStmt.java:209)
    at soot.jimple.spark.builder.MethodNodeFactory.handleStmt(MethodNodeFactory.java:150)
    at soot.jimple.spark.pag.MethodPAG.buildNormal(MethodPAG.java:224)
    at soot.jimple.spark.pag.MethodPAG.build(MethodPAG.java:186)
    at soot.jimple.spark.builder.ContextInsensitiveBuilder.handleClass(ContextInsensitiveBuilder.java:140)
    at soot.jimple.spark.builder.ContextInsensitiveBuilder.build(ContextInsensitiveBuilder.java:109)
    at soot.jimple.spark.SparkTransformer.internalTransform(SparkTransformer.java:101)
    at soot.SceneTransformer.transform(SceneTransformer.java:36)
    at soot.Transform.apply(Transform.java:105)
    at soot.RadioScenePack.internalApply(RadioScenePack.java:64)
    at soot.jimple.toolkits.callgraph.CallGraphPack.internalApply(CallGraphPack.java:61)
    at soot.Pack.apply(Pack.java:118)
        at ...

This is in the same context as #1724, i.e. with the following options:

-v -w -ignore-resolving-levels -cp my_classpath -prepend-classpath -process-dir subset_of_my_classpath -allow-phantom-refs -no-bodies-for-excluded -output-dir ./output -output-format c -ignore-resolution-errors -exclude [every_class_in_classpath_minus_process_dir] -keep-line-number -no-writeout-body-releasing -p cg.spark enabled:true vta:true -p jb use-original-names:true -p jb.sils enabled:false -p cg safe-newinstance:true -p cg all-reachable:true

[giltho]

Actually, I managed to reproduce it indepentently

This is running with

java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)

Files used to reproduce:

• log4j-api-2.12.1.jar, found in apache-log4j-2.12.1-bin.tar.gz which can be downloaded here : https://dlcdn.apache.org/logging/log4j/2.12.1/

Soot version:

trunk (https://github.com/soot-oss/soot/commit/9d88ffb66380b0b665d2f10810c89e462e72688f)

Command line:

-v -w -ignore-resolving-levels -cp log4j-api-2.12.1.jar -prepend-classpath -process-dir log4j-api-2.12.1.jar -allow-phantom-refs -output-dir ./output -ignore-resolution-errors -p cg.spark enabled:true vta:true -p cg all-reachable:true

Max Memory:

17560MB

Stack trace:

java.lang.RuntimeException: An error occurred while processing <META-INF.versions.9.org.apache.logging.log4j.util.Base64Util: java.lang.String encode(java.lang.String)> in callgraph
    at soot.jimple.spark.solver.OnFlyCallGraph.processReachables(OnFlyCallGraph.java:122)
    at soot.jimple.spark.solver.OnFlyCallGraph.build(OnFlyCallGraph.java:106)
    at soot.jimple.spark.builder.ContextInsensitiveBuilder.build(ContextInsensitiveBuilder.java:100)
    at soot.jimple.spark.SparkTransformer.internalTransform(SparkTransformer.java:101)
    at soot.SceneTransformer.transform(SceneTransformer.java:36)
    at soot.Transform.apply(Transform.java:105)
    at soot.RadioScenePack.internalApply(RadioScenePack.java:64)
    at soot.jimple.toolkits.callgraph.CallGraphPack.internalApply(CallGraphPack.java:61)
    at soot.Pack.apply(Pack.java:118)
    at soot.PackManager.runWholeProgramPacks(PackManager.java:619)
    at soot.PackManager.runPacksNormally(PackManager.java:500)
    at soot.PackManager.runPacks(PackManager.java:425)
    at soot.Main.run(Main.java:280)
    at soot.Main.main(Main.java:142)
Caused by: java.lang.NullPointerException
    at soot.jimple.spark.builder.MethodNodeFactory.caseStaticFieldRef(MethodNodeFactory.java:404)
    at soot.jimple.StaticFieldRef.apply(StaticFieldRef.java:83)
    at soot.jimple.spark.builder.MethodNodeFactory$1.caseAssignStmt(MethodNodeFactory.java:162)
    at soot.jimple.internal.JAssignStmt.apply(JAssignStmt.java:209)
    at soot.jimple.spark.builder.MethodNodeFactory.handleStmt(MethodNodeFactory.java:150)
    at soot.jimple.spark.pag.MethodPAG.buildNormal(MethodPAG.java:224)
    at soot.jimple.spark.pag.MethodPAG.build(MethodPAG.java:186)
    at soot.jimple.spark.solver.OnFlyCallGraph.processReachables(OnFlyCallGraph.java:116)
    ... 13 more

[lianah]

Just wanted to check if there are any updates on this. We are seeing this failure on 10% of the packages in one of the scans. Is the same issue as https://github.com/soot-oss/soot/issues/1724 ?

[linghuiluo]

Why do you need to set up all-reachable to be true?

[linghuiluo]

@giltho I took a look into the jar (a multi-release jar file) your provided. I think the problem came from the versioned classes under META-INF/versions/9. There are classes with the same signatures as the ones in the default package (see the screenshot below). Soot got confused by this. My suggestion to avoid this case is to remove methods of classes starting with signature "META-INF/versions/9" from the entry points. You used the option "all-reachable", which means all methods of application classes are set as entry points, including those from the META-INF/versions folder. I tried the code below before constructing the call graph and no exception anymore.

List<SootMethod> toRemove = new ArrayList<>();
for(SootMethod m: Scene.v().getEntryPoints()){
          if(m.getDeclaringClass().getName().startsWith("META-INF.version"))
             toRemove.add(m);
}
for(SootMethod m: toRemove ){
          Scene.v().getEntryPoints().remove(m);
}

[linghuiluo]

@anddann Since you are expert for Java 9, maybe you can answer my question. Does soot support versioned classes (Multi-release JAR files)? I dont think so. As far as I could see, classes under META-INF/version/9 get signatures with "META-INF.versions.9.someClasssSignature", but the methods and fields of those classes get "someClassSignature" in their signatures without "META-INF.version.9". This caused confusion in the resolution by soot.

[anddann]

@linghuiluo you are correct. Soot does not support versioned class (multi-release JAR files) for now. The JVM picks during runtime the classes it needs, e.g., JVM 11 picks classes in the folder version/11, and JVM 10 picks classes in the folder version/10. However, Soot does not have a specifier which version it should analyze, thus I've currently no quick idea how to fix that issue.

[linghuiluo]

@anddann thanks for the answer! @giltho @lianah I am afraid you have to filter out those versioned classes as I suggested above for now.

[lianah]

@linghuiluo thank you for the workaround. I will give it a try.

[lianah]

Hi @linghuiluo,

I gave the workaround a try but it only fixed the issue for this specific package. I seem to get the same error on other packages that do not involve any versioned classes:

Caused by: java.lang.RuntimeException: Unresolved type javax.ws.rs.client.ClientRequestFilter
        at soot.jimple.spark.pag.Node.<init>(Node.java:144)
        at soot.jimple.spark.pag.ValNode.<init>(ValNode.java:34)
        at soot.jimple.spark.pag.VarNode.<init>(VarNode.java:133)
        at soot.jimple.spark.pag.LocalVarNode.<init>(LocalVarNode.java:52)
        at soot.jimple.spark.pag.PAG.makeLocalVarNode(PAG.java:742)
        at soot.jimple.spark.builder.MethodNodeFactory.caseRet(MethodNodeFactory.java:297)
        at soot.jimple.spark.builder.MethodNodeFactory.setCurrentMethod(MethodNodeFactory.java:123)
        at soot.jimple.spark.builder.MethodNodeFactory.<init>(MethodNodeFactory.java:103)
        at soot.jimple.spark.pag.MethodPAG.<init>(MethodPAG.java:65)
        at soot.jimple.spark.pag.MethodPAG.v(MethodPAG.java:169)
        at soot.jimple.spark.builder.ContextInsensitiveBuilder.handleClass(ContextInsensitiveBuilder.java:139)
        at soot.jimple.spark.builder.ContextInsensitiveBuilder.build(ContextInsensitiveBuilder.java:109)
        at soot.jimple.spark.SparkTransformer.internalTransform(SparkTransformer.java:101)
        at soot.SceneTransformer.transform(SceneTransformer.java:36)
        at soot.Transform.apply(Transform.java:105)
        at soot.RadioScenePack.internalApply(RadioScenePack.java:64)
        at soot.jimple.toolkits.callgraph.CallGraphPack.internalApply(CallGraphPack.java:61)
        at soot.Pack.apply(Pack.java:118)
...

I managed to reproduce it with the attached .jar (Debug.jar.zip) containing just the following class build against https://mvnrepository.com/artifact/org.jboss.resteasy/jaxrs-api/3.0.7.Final

package debug;
import javax.ws.rs.client.ClientRequestFilter;

public class Main {

    public void foo(String[] args) {

        ClientRequestFilter filter = (ClientRequestFilter) clientRequestContext -> {
            System.out.println(clientRequestContext);
        };

The problem seems to be that ClientRequestFilter gets resolved to Dangling level which makes the TypeManager.isUnresolved return false and fail the assertion. After stepping through a debugger it seems that in the following line in the SootResolver.bringToHierarchyUnchecked the ClientRequestFilter class does not show up as a dependency of the debug.Main class:

Dependencies dependencies = is.resolve(sc);

it returns:

is.resolve(sc)
IInitialResolver$Dependencies@6600
typesToHierarchy: HashSet@6602 size=0
typesToSignature: HashSet@6603 size=8
0: RefType@6613 "java.io.PrintStream"
1: RefType@6614 "java.lang.invoke.MethodHandles$Lookup"
2: RefType@6615 "javax.ws.rs.client.ClientRequestContext"
3: RefType@6616 "java.lang.System"
4: VoidType@6617 "void"
5: RefType@6618 "java.lang.Object"
6: ArrayType@6619 "java.lang.String[]"
7: RefType@6620 "java.io.IOException"

I ran Soot with the following CallGraph options:

        sootOptions.setPhaseOption("cg.spark", "enabled:true");
        sootOptions.setPhaseOption("cg.spark", "vta:true");
        sootOptions.setPhaseOption("cg", "safe-newinstance:true");
        sootOptions.setPhaseOption("cg", "all-reachable:false");

and the soot.options.Options class has the following values:

oot.options.Options.v()
Options@6633
allow_cg_errors: false
allow_phantom_elms: false
allow_phantom_refs: true
android_api_version: -1
android_jars: ""
app: false
ast_metrics: false
check_init_throw_analysis: 0
classes: LinkedList@6635 size=0
coffi: false
debug: false
debug_resolver: false
derive_java_version: true
drop_bodies_after_load: true
dump_body: null
dump_cfg: null
dynamic_class: null
dynamic_dir: null
dynamic_package: null
exclude: ArrayList@6636 size=0
field_type_mismatches: 0
force_android_jar: ""
force_overwrite: false
full_resolver: false
gzip: false
help: false
hierarchy_dirs: false
ignore_classpath_errors: false
ignore_resolution_errors: true
ignore_resolving_levels: true
include: null
include_all: false
interactive_mode: false
j2me: false
jasmin_backend: false
java_version: 12
keep_line_number: true
keep_offset: false
main_class: ""
no_bodies_for_excluded: true
no_output_inner_classes_attribute: false
no_output_source_file_attribute: false
no_writeout_body_releasing: true
num_threads: -1
oaat: false
omit_excepting_unit_edges: false
on_the_fly: false
options: LinkedList@6637 size=0
output_dir: "path/to/ds_output/sootOutput"
output_format: 1
output_jar: false
permissive_resolving: false
phase_help: null
phase_list: false
plugin: null
polyglot: false
prepend_classpath: true
print_tags_in_output: false
process_dir: ArrayList@6639 size=1
0: "/path/to/reproducer/src/Debug.jar"
process_jar_dir: null
process_multiple_dex: false
search_dex_in_archives: false
show_exception_dests: true
soot_classpath: "VIRTUAL_FS_FOR_JDK:/path/to/reproducer/src/Debug.jar"
soot_modulepath: ""
src_prec: 0
subtract_gc: false
throw_analysis: 0
time: false
unfriendly_mode: false
validate: false
verbose: true
version: false
via_grimp: false
via_shimple: false
weak_map_structures: false
whole_program: true
whole_shimple: false
write_local_annotations: false
wrong_staticness: 0
xml_attributes: false

[linghuiluo]

@lianah Are you sure about the option "all-reachable:false"? I could not reproduce it with this option disabled, but with it enabled. The problem is indeed due to resolution level of the type, It should not throw the exception if the ignore_resolution_errors is enabled. I made this change https://github.com/soot-oss/soot/commit/f935dec864b3ed9920ba76062e81dc42f748ce34 This should make the exception away.