[Bug] Jimple2cpg stack trace during spring web jar import

[itsacoderepo]

Describe the bug Importing this jar Spring-Web 5.2.2 is causing a stack trace:

Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on -Dswing.aatext=true
2023-05-06 19:40:52.146 WARN AstCreationPass: Unexpected runtime exception while parsing method body! Will stub the method 'org.springframework.http.client.Netty4ClientHttpRequest.executeInternal:org.springframework.util.concurrent.ListenableFuture(org.springframework.http.HttpHeaders)''
java.lang.RuntimeException: Failed to apply jb to <org.springframework.http.client.Netty4ClientHttpRequest: org.springframework.util.concurrent.ListenableFuture executeInternal(org.springframework.http.HttpHeaders)>
    at soot.asm.AsmMethodSource.getBody(AsmMethodSource.java:2281) ~[org.soot-oss.soot-4.4.1.jar:?]

[DavidBakerEffendi]

In the stack trace, it is suggested to do:

Scene.v().addBasicClass(io.netty.channel.ChannelFutureListener,HIERARCHY);
Otherwise, try whole-program mode (-w).

Neither of them fix the issue.

I've had this issue before, but I've never been able to find a fix for this.

[DavidBakerEffendi]

@evilpan Have you ever managed to fix this kind of issue?

[evilpan]

Sorry that I missed the metion. For this case whole program mode (-W) should work, but somehow it throws another exception:

$ java -cp soot-4.4.1-jar-with-dependencies.jar soot.Main -W -process-path spring-web-5.2.2.RELEASE.jar -f J -d out -allow-phantom-refs
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
Soot started on Thu Jun 01 10:50:08 CST 2023
java.util.ConcurrentModificationException
        at soot.util.HashChain$LinkIterator.hasNext(HashChain.java:582)
        at soot.jimple.toolkits.invoke.StaticInliner.computeAverageMethodSizeAndSaveOriginalSizes(StaticInliner.java:149)
        at soot.jimple.toolkits.invoke.StaticInliner.internalTransform(StaticInliner.java:74)
        at soot.SceneTransformer.transform(SceneTransformer.java:36)
        at soot.Transform.apply(Transform.java:105)
        at soot.ScenePack.internalApply(ScenePack.java:37)
        at soot.Pack.apply(Pack.java:118)
        at soot.PackManager.runWholeProgramPacks(PackManager.java:621)
        at soot.PackManager.runPacksNormally(PackManager.java:500)
        at soot.PackManager.runPacks(PackManager.java:425)
        at soot.Main.run(Main.java:266)
        at soot.Main.main(Main.java:142)

And this was tagged as bug in soot https://github.com/soot-oss/soot/issues/1206 without a fix yet. Maybe I can create some tests with sootup to see if it's solved in their successor.

[DavidBakerEffendi]

@evilpan Thanks! If you can show that SootUp is more stable, then maybe I'll need to start a replacement branch

[evilpan]

Hi @DavidBakerEffendi According to my experiment, the exception in jimple2cpg is caused by these two lines https://github.com/joernio/joern/blob/master/joern-cli/frontends/jimple2cpg/src/main/scala/io/joern/jimple2cpg/Jimple2Cpg.scala#L178-L180

// output jimple
Options.v().set_output_format(Options.output_format_jimple)
Options.v().set_output_dir(ProgramHandlingUtil.getUnpackingDir.toString)

After commenting out these line, parsing should not throw any exceptions. Most of time people don't need to generate jimple files through jimple2cpg and we don't have any options to toggle this feature, so I think comment out these output option is a good idea for now.

[DavidBakerEffendi]

Thanks, @evilpan! I'll follow up with this soon, there are some other issues with jimple2cpg I need to look at, but that would fix a bunch of other legacy issues.

[DavidBakerEffendi]

@evilpan seems like I still get the stacktrace above when the above code is commented out. But perhaps this PR sheds some light on the issue https://github.com/joernio/joern/pull/2839

[evilpan]

That's weird, I just pulled the latest code and find out that jimple2cpg can successfully generate cpg of spring-web.jar without any error. Here's what I did:

$ git pull && sbt stage
$ ./joern-cli/frontends/jimple2cpg/jimple2cpg spring-web-5.2.2.RELEASE.jar -o spring.cpg
$ ls -l spring.cpg
-rw-r--r-- 1 pan staff 17235968 Jun 11 16:50 spring.cpg
$ ./joern
Compiling /Users/pan/Tools/joern/(console)

     ██╗ ██████╗ ███████╗██████╗ ███╗   ██╗
     ██║██╔═══██╗██╔════╝██╔══██╗████╗  ██║
     ██║██║   ██║█████╗  ██████╔╝██╔██╗ ██║
██   ██║██║   ██║██╔══╝  ██╔══██╗██║╚██╗██║
╚█████╔╝╚██████╔╝███████╗██║  ██║██║ ╚████║
 ╚════╝  ╚═════╝ ╚══════╝╚═╝  ╚═╝╚═╝  ╚═══╝
Version: 1.2.12
Type `help` or `browse(help)` to begin

joern> importCpg("spring.cpg")
Creating project `spring.cpg` for CPG at `spring.cpg`
Creating working copy of CPG to be safe
Loading base CPG from: /Users/pan/Tools/joern/workspace/spring.cpg/cpg.bin.tmp
Adding default overlays to base CPG
The graph has been modified. You may want to use the `save` command to persist changes to disk.  All changes will also be saved collectively on exit
The graph has been modified. You may want to use the `save` command to persist changes to disk.  All changes will also be saved collectively on exit
res0: Option[Cpg] = Some(value = Cpg (Graph [298779 nodes]))

joern> cpg.typeDecl.size
res1: Int = 2248

I've tried on both MacOS and Ubuntu(22.04.1 LTS) and got similar results.

[DavidBakerEffendi]

@evilpan It will be successful, but the error/warning describes when a method body will be empty - it will continue generating the CPG though. You can get that by using importCode on the jar directly.

[evilpan]

Hi @DavidBakerEffendi, After some digging, I may found the root cause of this issue.

In method Netty4ClientHttpRequest.executeInternal, there's a implicit conversion from lambda expression to functional interface:

@Override
protected ListenableFuture<ClientHttpResponse> executeInternal(final HttpHeaders headers) throws IOException {
    final SettableListenableFuture<ClientHttpResponse> responseFuture = new SettableListenableFuture<>();

    ChannelFutureListener connectionListener = future -> {
        if (future.isSuccess()) {
            Channel channel = future.channel();
            channel.pipeline().addLast(new RequestExecuteHandler(responseFuture));
            FullHttpRequest nettyRequest = createFullHttpRequest(headers);
            channel.writeAndFlush(nettyRequest);
        }
        else {
            responseFuture.setException(future.cause());
        }
    };
  // ...
}

However the io.netty.channel.ChannelFutureListener interface is not contained in spring-web.jar so it's resolving level is DANGLING. That's why we still get this exception even when whole program mode is enabled.

I've also fire an issue on Soot https://github.com/soot-oss/soot/issues/1975 and hope they can deal with this conversion.

[DavidBakerEffendi]

I definitely think the error message from Soot should be much more helpful, but it may require the classes to be dynamically loaded, as is now possible via this config: https://github.com/joernio/joern/pull/2839