giltho commented 3 years ago

Hi!

This might, or might not be an actual bug report, it is very difficult to assess. We're using Soot as a library to extract some callgraphs.

In particular, while running PackManager.v().getPack("cg").apply(), the following exception gets raised.

Exception in thread "main" java.lang.RuntimeException: Unresolved type java.lang.Object
        at soot.jimple.spark.pag.Node.<init>(Node.java:144)
        at soot.jimple.spark.pag.ValNode.<init>(ValNode.java:34)
        at soot.jimple.spark.pag.VarNode.<init>(VarNode.java:133)
        at soot.jimple.spark.pag.LocalVarNode.<init>(LocalVarNode.java:52)
        at soot.jimple.spark.pag.PAG.makeLocalVarNode(PAG.java:742)
        at soot.jimple.spark.builder.MethodNodeFactory.caseCastExpr(MethodNodeFactory.java:317)
        at soot.jimple.internal.AbstractCastExpr.apply(AbstractCastExpr.java:128)
        at soot.jimple.spark.builder.MethodNodeFactory$1.caseAssignStmt(MethodNodeFactory.java:162)
        at soot.jimple.internal.JAssignStmt.apply(JAssignStmt.java:242)
        at soot.jimple.spark.builder.MethodNodeFactory.handleStmt(MethodNodeFactory.java:150)
        at soot.jimple.spark.pag.MethodPAG.buildNormal(MethodPAG.java:224)
        at soot.jimple.spark.pag.MethodPAG.build(MethodPAG.java:186)
        at soot.jimple.spark.builder.ContextInsensitiveBuilder.handleClass(ContextInsensitiveBuilder.java:140)
        at soot.jimple.spark.builder.ContextInsensitiveBuilder.build(ContextInsensitiveBuilder.java:109)
        at soot.jimple.spark.SparkTransformer.internalTransform(SparkTransformer.java:101)
        at soot.SceneTransformer.transform(SceneTransformer.java:36)
        at soot.Transform.apply(Transform.java:102)
        at soot.RadioScenePack.internalApply(RadioScenePack.java:68)
        at soot.jimple.toolkits.callgraph.CallGraphPack.internalApply(CallGraphPack.java:58)
        at soot.Pack.apply(Pack.java:117)
        // REDACTED

I serialized the Options right before calling that, and I obtain:

{
  "coffi": false,
  "jasmin_backend": false,
  "help": false,
  "phase_list": false,
  "version": false,
  "verbose": true,
  "interactive_mode": false,
  "unfriendly_mode": false,
  "app": false,
  "whole_program": true,
  "whole_shimple": false,
  "on_the_fly": false,
  "validate": false,
  "debug": false,
  "debug_resolver": false,
  "ignore_resolving_levels": true,
  "weak_map_structures": false,
  "soot_classpath": "QUITE A HUGE CLASSPATH, INCLUDING pdfbox-2.0.15.jar",
  "soot_modulepath": "",
  "prepend_classpath": true,
  "ignore_classpath_errors": false,
  "process_multiple_dex": false,
  "search_dex_in_archives": false,
  "process_dir": [
    "/path/to/pdfbox-2.0.15.jar"
  ],
  "derive_java_version": true,
  "oaat": false,
  "android_jars": "",
  "force_android_jar": "",
  "android_api_version": -1,
  "ast_metrics": false,
  "src_prec": 0,
  "full_resolver": false,
  "allow_phantom_refs": true,
  "allow_phantom_elms": false,
  "allow_cg_errors": false,
  "no_bodies_for_excluded": true,
  "j2me": false,
  "main_class": "",
  "polyglot": false,
  "permissive_resolving": false,
  "drop_bodies_after_load": true,
  "output_dir": "output",
  "output_format": 1,
  "java_version": 12,
  "output_jar": false,
  "hierarchy_dirs": false,
  "xml_attributes": false,
  "print_tags_in_output": false,
  "no_output_source_file_attribute": false,
  "no_output_inner_classes_attribute": false,
  "show_exception_dests": true,
  "gzip": false,
  "force_overwrite": false,
  "wrong_staticness": 0,
  "field_type_mismatches": 0,
  "via_grimp": false,
  "via_shimple": false,
  "throw_analysis": 0,
  "check_init_throw_analysis": 0,
  "omit_excepting_unit_edges": false,
  "ignore_resolution_errors": true,
  "exclude": [ A LOT OF THINGS HERE BUT NO java.lang.Object ]
}

It is interesting to note that I have done the exact same experiment but with only pdfbox and its dependencies in the classpath (and also a LOT less excludes) and the exception does not get thrown.

Also, it is not the first time we encounter Unresolved type xxxxx while building a VTA callgraph, but the other example were not as striking as java.lang.Object.

Would you have any idea of the source of that issue? Unfortunately this project is not open source, so I can't just redirect you to it, but I'd be glad to answer any of your questions!

Thanks

swissiety commented 3 years ago

It seems as if the java runtime library (which contains the corresponding java.lang.Object) is not found by Soot. Is the path to the rt.jar found on the huge classpath ? Or as you specified java_version 12: is VIRTUAL_FS_FOR_JDK added to the classpath to access the runtime library for >=java9 (see https://github.com/soot-oss/soot#example-configurations-java-8-java--9-classpath-java--9-modulepath) ?

giltho commented 3 years ago

The huge classpath does not contain rt.jar, but prepend_classpath is set to true (this is with jdk8). That should be enough? In any case, it looks like there is a discrepency, because running the exact same application on the same code, but with a different classpath (which also does not contain rt.jar) & exclude field will not fail.

giltho commented 3 years ago

Ok I've tried more things here, and, in the debugger, I can confirm that right before calling PackManager.v().getPack("cg").apply(), (during which the Exception in thread "main" java.lang.RuntimeException: Unresolved type java.lang.Object happens), java.lang.Object is in the scene (soot.Scene.v().classes) I can't understand what causes this unresolved type issue, where should the class be loaded for that not to happen?

Moreover, the error is happening here, in the Node constructor, because Type.isResolved(type) returned false. At that moment, in the debugger, I can see that type has the following value: And the value spells out java.lang.Object

giltho commented 3 years ago

Hi! So I managed to reproduce it in command-line only and without company-specific tooling / non-public source necessary. Here's the message template generated by soot on error:

Error

Steps to reproduce: 1.) download the two jar files (pdfbox-2.0.15.jar and TestPdfBoxSoot.jar) 2.) Run the command line with given arguments

Files used to reproduce:

pdfbox-2.0.15.jar from the apache archive
TestPdfBoxSoot.jar from this release of this repo

Soot version:

trunk

Command line:

-v -w -ignore-resolving-levels -cp /path/to/TestPdfBoxSoot.jar:/path/to/pdfbox-2.0.15.jar -prepend-classpath -process-dir /path/to/pdfbox-2.0.15.jar -allow-phantom-refs -no-bodies-for-excluded -output-dir ./output -output-format J -ignore-resolution-errors -exclude com.amazon.testPdfBoxSoot.Main -exclude com.amazon.testPdfBoxSoot.package-info -keep-line-number -no-writeout-body-releasing -p cg.spark enabled:true vta:true -p jb use-original-names:true -p jb.sils enabled:false -p cg safe-newinstance:true -p cg all-reachable:true

Max Memory:

17560MB

Stack trace:

java.lang.RuntimeException: An error occurred while processing <org.apache.pdfbox.pdmodel.graphics.image.LosslessFactory$PredictorEncoder: org.apache.pdfbox.pdmodel.graphics.image.PDImageXObject encode()> in callgraph
    at soot.jimple.spark.solver.OnFlyCallGraph.processReachables(OnFlyCallGraph.java:122)
    at soot.jimple.spark.solver.OnFlyCallGraph.build(OnFlyCallGraph.java:106)
    at soot.jimple.spark.builder.ContextInsensitiveBuilder.build(ContextInsensitiveBuilder.java:100)
    at soot.jimple.spark.SparkTransformer.internalTransform(SparkTransformer.java:101)
    at soot.SceneTransformer.transform(SceneTransformer.java:36)
    at soot.Transform.apply(Transform.java:105)
    at soot.RadioScenePack.internalApply(RadioScenePack.java:64)
    at soot.jimple.toolkits.callgraph.CallGraphPack.internalApply(CallGraphPack.java:61)
    at soot.Pack.apply(Pack.java:118)
    at soot.PackManager.runWholeProgramPacks(PackManager.java:619)
    at soot.PackManager.runPacksNormally(PackManager.java:500)
    at soot.PackManager.runPacks(PackManager.java:425)
    at soot.Main.run(Main.java:280)
    at soot.Main.main(Main.java:142)
Caused by: java.lang.RuntimeException: Unresolved type java.lang.Object
    at soot.jimple.spark.pag.Node.<init>(Node.java:144)
    at soot.jimple.spark.pag.ValNode.<init>(ValNode.java:34)
    at soot.jimple.spark.pag.VarNode.<init>(VarNode.java:133)
    at soot.jimple.spark.pag.LocalVarNode.<init>(LocalVarNode.java:52)
    at soot.jimple.spark.pag.PAG.makeLocalVarNode(PAG.java:742)
    at soot.jimple.spark.builder.MethodNodeFactory.caseCastExpr(MethodNodeFactory.java:322)
    at soot.jimple.internal.AbstractCastExpr.apply(AbstractCastExpr.java:136)
    at soot.jimple.spark.builder.MethodNodeFactory$1.caseAssignStmt(MethodNodeFactory.java:162)
    at soot.jimple.internal.JAssignStmt.apply(JAssignStmt.java:209)
    at soot.jimple.spark.builder.MethodNodeFactory.handleStmt(MethodNodeFactory.java:150)
    at soot.jimple.spark.pag.MethodPAG.buildNormal(MethodPAG.java:224)
    at soot.jimple.spark.pag.MethodPAG.build(MethodPAG.java:186)
    at soot.jimple.spark.solver.OnFlyCallGraph.processReachables(OnFlyCallGraph.java:116)
    ... 13 more

Note:

The error does not happen if the parameter -p cg all-reachable:true is removed

giltho commented 3 years ago

Ok, so I just did so more tests (sorry for spamming here, I should have done that earlier) but actually, it doesn't need all of that. In particular, using TestPdfBoxSoot.jar is useless here, it can be reproduced using only pdfbox, and fewer options.

Command line:

-v -w -ignore-resolving-levels -cp "/path/to/pdfbox-2.0.15.jar" -prepend-classpath -process-dir "/path/to/pdfbox-2.0.15.jar" -allow-phantom-refs -output-dir ./output -ignore-resolution-errors -p cg.spark enabled:true vta:true -p cg all-reachable:true

giltho commented 3 years ago

See #1739

giltho commented 3 years ago

So actually, this might not be entirely fixed... I'm still getting a very similar trace, although it now happens a lot less often, but it still does happen on some packages.

Unfortunately, this time, it only happens on internal packages that I am not able to share, here's the trace

Exception in thread "main" java.lang.RuntimeException: Unresolved type [REDACTED]
    at soot.jimple.spark.pag.Node.<init>(Node.java:144)
    at soot.jimple.spark.pag.ValNode.<init>(ValNode.java:34)
    at soot.jimple.spark.pag.VarNode.<init>(VarNode.java:133)
    at soot.jimple.spark.pag.LocalVarNode.<init>(LocalVarNode.java:52)
    at soot.jimple.spark.pag.PAG.makeLocalVarNode(PAG.java:733)
    at soot.jimple.spark.builder.MethodNodeFactory.caseLocal(MethodNodeFactory.java:343)
    at soot.jimple.internal.JimpleLocal.apply(JimpleLocal.java:125)
    at soot.jimple.spark.builder.MethodNodeFactory$1.caseAssignStmt(MethodNodeFactory.java:160)
    at soot.jimple.internal.JAssignStmt.apply(JAssignStmt.java:209)
    at soot.jimple.spark.builder.MethodNodeFactory.handleStmt(MethodNodeFactory.java:150)
    at soot.jimple.spark.pag.MethodPAG.buildNormal(MethodPAG.java:224)
    at soot.jimple.spark.pag.MethodPAG.build(MethodPAG.java:186)
    at soot.jimple.spark.builder.ContextInsensitiveBuilder.handleClass(ContextInsensitiveBuilder.java:140)
    at soot.jimple.spark.builder.ContextInsensitiveBuilder.build(ContextInsensitiveBuilder.java:109)
    at soot.jimple.spark.SparkTransformer.internalTransform(SparkTransformer.java:101)
    at soot.SceneTransformer.transform(SceneTransformer.java:36)
    at soot.Transform.apply(Transform.java:105)
    at soot.RadioScenePack.internalApply(RadioScenePack.java:64)
    at soot.jimple.toolkits.callgraph.CallGraphPack.internalApply(CallGraphPack.java:61)
    at soot.Pack.apply(Pack.java:118)
    ...

Note that this is now in a different case (caseLocal instead of caseCastExpr)

soot-oss / soot

Unresolved type java.lang.object while building a VTA callgraph. #1724

Error

Note: