the script "joern/graph-for-funcs.sc" cannot run successfully, how to solve it?

Trunto commented 4 years ago

enviroment:

computer: Linux version 4.4.0-124-generic (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) )
java version "1.8.0_251",Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Scala code runner version 2.13.3

the error log output: j`` oern> cpg.runScript("/home/bfs/cct/devign/joern/graph-for-funcs.sc").toString() Compiling (synthetic)/ammonite/predef/interpBridge.sc Compiling (synthetic)/ammonite/predef/replBridge.sc Compiling (synthetic)/ammonite/predef/sourceBridge.sc Compiling (synthetic)/ammonite/predef/frontEndBridge.sc Compiling (synthetic)/ammonite/predef/DefaultPredef.sc Compiling (synthetic)/ammonite/predef/ArgsPredef.sc Compiling /home/bfs/cct/scala/sbt/(console) Compiling /home/bfs/cct/devign/joern/graph-for-funcs.sc graph-for-funcs.sc:86: **value filterOnEnd is not a member of java.util.Iterator[overflowdb.Node]** method.out(EdgeTypes.CONTAINS).filterOnEnd(_.isInstanceOf[nodes.CfgNode]).cast[nodes.CfgNode] ^ graph-for-funcs.sc:92: **value hasLabel is not a member of java.util.Iterator[overflowdb.Node]** possible cause: maybe a semicolon is missing beforevalue hasLabel`? .hasLabel(NodeTypes.BLOCK) ^ graph-for-funcs.sc:97: value hasLabel is not a member of java.util.Iterator[overflowdb.Node] val source = new NodeSteps(method.out(EdgeTypes.CONTAINS).hasLabel(NodeTypes.CALL).cast[nodes.Call]).nameNot(".*").dedup ^ java.lang.RuntimeException: Compilation Failed io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:52) cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:229) cats.effect.internals.IORunLoop$.step(IORunLoop.scala:229) cats.effect.IO.unsafeRunTimed(IO.scala:320) cats.effect.IO.unsafeRunSync(IO.scala:239) io.shiftleft.console.scripting.ScriptManager.runScript(ScriptManager.scala:130) io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:64) io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:54) ammonite.$sess.cmd7$.(cmd7.sc:1)



Have you encountered this problem ? and do you know where the problem is? I look forward to your help to provide ideas to solve the problem. 
Thank you very much

epicosy commented 4 years ago

It is most probable that Joern's implementation was changed with something different. I just extended their script for combining the AST, CGF, and PDG for each method to include the file path. I did a diff between the script in my repository (on the left) and their current version of the script (on the right) here. Was it helpful?

Trunto commented 4 years ago

It is most probable that Joern's implementation was changed with something different. I just extended their script for combining the AST, CGF, and PDG for each method to include the file path. I did a diff between the script in my repository (on the left) and their current version of the script (on the right) here. Was it helpful?

Thank you very much! The error comes from the cfgChildren generation ( line 80 of graph-for-funcs.sc in your repository ), when i change it to the current version of the script, it worked! Thank you again for your answer

QiuJYWX commented 4 years ago

Hi @Trunto @epicosy When I ran python main.py -c, I came across the issue: Traceback (most recent call last): File "main.py", line 157, in main() File "main.py", line 146, in main create_task() File "main.py", line 61, in create_task graphs = prepare.json_process(PATHS.cpg, json_file) File "/home/qjy/Desktop/devign/src/prepare/cpg_generator.py", line 79, in json_process container = [graph_indexing(graph) for graph in cpg_json["functions"] if graph["file"] != "N/A"] File "/home/qjy/Desktop/devign/src/prepare/cpg_generator.py", line 79, in container = [graph_indexing(graph) for graph in cpg_json["functions"] if graph["file"] != "N/A"] KeyError: 'file'

Anyone one can provide me some advice?

epicosy commented 4 years ago

@Trunto @QiuJYWX be aware that in the graph-for-funcs.sc script there is the extra "file: String" attribute which is added to the JSON results that are used in the main.py scripts. If you update the graph-for-funcs.sc script don't forget to add it. I'll open an issue for that.

QiuJYWX commented 4 years ago

Hi @epicosy , Thanks for your quickly and kindly reply. BTW, in which line should I add the code? Would you please share the updated graph-for-funcs.sc file? Thanks:)

epicosy commented 4 years ago

@Trunto @QiuJYWX just updated it, here

QiuJYWX commented 4 years ago

Hi @epicosy , great ! Thanks for the update

Sos-sc2 commented 4 years ago

嗨@epicosy，太好了！感谢更新

请问你第一个create task完成了嘛？我一直有一些问题

faysalhossain2007 commented 2 years ago

I am facing the following error with Joern 1.1.55. Can anyone tell me which is the correct version that they were successful in running devign tool?

graph-for-funcs.sc:50: not found: type OdbEdge
implicit val encodeEdge: Encoder[OdbEdge] =
                                 ^
graph-for-funcs.sc:51: not found: type OdbEdge
  (edge: OdbEdge) =>
         ^
graph-for-funcs.sc:63: could not find implicit value for parameter encoder: io.circe.Encoder[overflowdb.Edge]
        Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))),
                                                                                      ^
graph-for-funcs.sc:81: not found: type Vertex
      val methodVertex: Vertex = method //TODO MP drop as soon as we have the remainder of the below in ODB graph api
                        ^
java.lang.RuntimeException: Compilation Failed
  io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:50)
  io.shiftleft.console.scripting.AmmoniteExecutor$$Lambda$1798/0x0000000000000000.apply(Unknown Source)
  cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:229)
  cats.effect.internals.IORunLoop$.step(IORunLoop.scala:229)
  cats.effect.IO.unsafeRunTimed(IO.scala:320)
  cats.effect.IO.unsafeRunSync(IO.scala:239)
  io.shiftleft.console.scripting.ScriptManager.runScript(ScriptManager.scala:130)
  io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:64)
  io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:54)
  ammonite.$sess.cmd4$.<clinit>(cmd4.sc:1)

The following are my java configurations:

openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
Eclipse OpenJ9 VM AdoptOpenJDK-11.0.11+9 (build openj9-0.26.0, JRE 11 Linux amd64-64-Bit Compressed References 20210421_975 (JIT enabled, AOT enabled)
OpenJ9   - b4cc246d9
OMR      - 162e6f729
JCL      - 7796c80419 based on jdk-11.0.11+9)

Woffee commented 2 years ago

I am facing the following error with Joern 1.1.55. Can anyone tell me which is the correct version that they were successful in running devign tool?

graph-for-funcs.sc:50: not found: type OdbEdge
implicit val encodeEdge: Encoder[OdbEdge] =
                                 ^
graph-for-funcs.sc:51: not found: type OdbEdge
  (edge: OdbEdge) =>
         ^
graph-for-funcs.sc:63: could not find implicit value for parameter encoder: io.circe.Encoder[overflowdb.Edge]
        Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))),
                                                                                      ^
graph-for-funcs.sc:81: not found: type Vertex
      val methodVertex: Vertex = method //TODO MP drop as soon as we have the remainder of the below in ODB graph api
                        ^
java.lang.RuntimeException: Compilation Failed
  io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:50)
  io.shiftleft.console.scripting.AmmoniteExecutor$$Lambda$1798/0x0000000000000000.apply(Unknown Source)
  cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:229)
  cats.effect.internals.IORunLoop$.step(IORunLoop.scala:229)
  cats.effect.IO.unsafeRunTimed(IO.scala:320)
  cats.effect.IO.unsafeRunSync(IO.scala:239)
  io.shiftleft.console.scripting.ScriptManager.runScript(ScriptManager.scala:130)
  io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:64)
  io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:54)
  ammonite.$sess.cmd4$.<clinit>(cmd4.sc:1)

The following are my java configurations:

openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
Eclipse OpenJ9 VM AdoptOpenJDK-11.0.11+9 (build openj9-0.26.0, JRE 11 Linux amd64-64-Bit Compressed References 20210421_975 (JIT enabled, AOT enabled)
OpenJ9   - b4cc246d9
OMR      - 162e6f729
JCL      - 7796c80419 based on jdk-11.0.11+9)

I used the Joern version v1.0.105: https://github.com/joernio/joern/releases/download/v1.0.105/joern-cli.zip

and graph-for-funcs.sc in this commit: https://github.com/epicosy/devign/blob/e7377dd49982d531707172d941a8933f6acd53f0/joern/graph-for-funcs.sc

They work for me.

ashrafflh commented 2 years ago

Here is an updated script that should work for the latest releases of Joern. graph-for-funcs.txt

Ningke-Li commented 1 year ago

Here is an updated script that should work for the latest releases of Joern. graph-for-funcs.txt

Hi sir, I use your graph-for-funcs.sc with Joern V1.1.1289 and got the following error response. Do you have any idea which place is wrong? Thanks a lot.

joern/scripts/graph-for-funcs.sc:66: value l is not a member of java.util.Iterator[overflowdb.Edge] Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))),

joern/scripts/graph-for-funcs.sc:92: value hasLabel is not a member of java.util.Iterator[overflowdb.Node] possible cause: maybe a semicolon is missing before value hasLabel? .hasLabel(NodeTypes.BLOCK)

joern/scripts/graph-for-funcs.sc:97: value hasLabel is not a member of java.util.Iterator[overflowdb.Node] val source = new Traversal(method.out(EdgeTypes.CONTAINS).hasLabel(NodeTypes.CALL).cast[nodes.Call]).nameNot(".*").dedup

duhang1996 commented 1 year ago

Here is an updated script that should work for the latest releases of Joern. graph-for-funcs.txt

Hi sir, I use your graph-for-funcs.sc with Joern V1.1.1289 and got the following error response. Do you have any idea which place is wrong? Thanks a lot.

joern/scripts/graph-for-funcs.sc:66: value l is not a member of java.util.Iterator[overflowdb.Edge] Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))),

joern/scripts/graph-for-funcs.sc:66: value l is not a member of java.util.Iterator[overflowdb.Edge] Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))),

joern/scripts/graph-for-funcs.sc:92: value hasLabel is not a member of java.util.Iterator[overflowdb.Node] possible cause: maybe a semicolon is missing before value hasLabel? .hasLabel(NodeTypes.BLOCK)

joern/scripts/graph-for-funcs.sc:97: value hasLabel is not a member of java.util.Iterator[overflowdb.Node] val source = new Traversal(method.out(EdgeTypes.CONTAINS).hasLabel(NodeTypes.CALL).cast[nodes.Call]).nameNot(".*").dedup

这个问题解决了没

664730 commented 7 months ago

May I ask what's wrong with the killed here? It used to be useful, but now it can't generate a JSON file.

epicosy / devign

the script "joern/graph-for-funcs.sc" cannot run successfully, how to solve it? #3