Mismatch methods in graph-for-funcs.sc, may need update?

[zhaohan-xi]

Hi Dear Author,

Here is a issue about running graph-for-funcs.sc. There seems to have a lot of changes on joern methods and scalars, thus I try to modify your code to fit into the most recent one, however, here are two problems that cannot be solved yet whatever I try. Both are about the lacking methods in io.shiftleft.semanticcpg.language.NodeSteps. I checked the official repository (https://github.com/ShiftLeftSecurity/codepropertygraph) and go to codepropertygraph/semanticcpg/src/main/scala/io/shiftleft/semanticcpg/language/NodeSteps.scala, but cannot find correct methods for these two errors in line-93 and 94. Your help would be necessary for that. Thanks!

Compiling /home/zxx5113/devign/joern/graph-for-funcs.sc
graph-for-funcs.sc:93: value evalType is not a member of io.shiftleft.semanticcpg.language.NodeSteps[io.shiftleft.codepropertygraph.generated.nodes.Local]
      val sink = local.evalType(".*").referencingIdentifiers.dedup
                       ^
graph-for-funcs.sc:94: value nameNot is not a member of io.shiftleft.semanticcpg.language.NodeSteps[io.shiftleft.codepropertygraph.generated.nodes.Call]
      val source = new NodeSteps(methodVertex.out(EdgeTypes.CONTAINS).hasLabel(NodeTypes.CALL).cast[nodes.Call]).nameNot("<operator>.*").dedup
                                                                                                                 ^
java.lang.RuntimeException: Compilation Failed
  io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:50)
  cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:229)
  cats.effect.internals.IORunLoop$.step(IORunLoop.scala:229)
  cats.effect.IO.unsafeRunTimed(IO.scala:320)
  cats.effect.IO.unsafeRunSync(IO.scala:239)
  io.shiftleft.console.scripting.ScriptManager.runScript(ScriptManager.scala:130)
  io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:64)
  io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:54)
  ammonite.$sess.cmd4$.<clinit>(cmd4.sc:1)

Bye!

[GoneZ5]

Hi Dear Author,

Here is a issue about running graph-for-funcs.sc. There seems to have a lot of changes on joern methods and scalars, thus I try to modify your code to fit into the most recent one, however, here are two problems that cannot be solved yet whatever I try. Both are about the lacking methods in io.shiftleft.semanticcpg.language.NodeSteps. I checked the official repository (https://github.com/ShiftLeftSecurity/codepropertygraph) and go to codepropertygraph/semanticcpg/src/main/scala/io/shiftleft/semanticcpg/language/NodeSteps.scala, but cannot find correct methods for these two errors in line-93 and 94. Your help would be necessary for that. Thanks!

Compiling /home/zxx5113/devign/joern/graph-for-funcs.sc
graph-for-funcs.sc:93: value evalType is not a member of io.shiftleft.semanticcpg.language.NodeSteps[io.shiftleft.codepropertygraph.generated.nodes.Local]
      val sink = local.evalType(".*").referencingIdentifiers.dedup
                       ^
graph-for-funcs.sc:94: value nameNot is not a member of io.shiftleft.semanticcpg.language.NodeSteps[io.shiftleft.codepropertygraph.generated.nodes.Call]
      val source = new NodeSteps(methodVertex.out(EdgeTypes.CONTAINS).hasLabel(NodeTypes.CALL).cast[nodes.Call]).nameNot("<operator>.*").dedup
                                                                                                                 ^
java.lang.RuntimeException: Compilation Failed
  io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:50)
  cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:229)
  cats.effect.internals.IORunLoop$.step(IORunLoop.scala:229)
  cats.effect.IO.unsafeRunTimed(IO.scala:320)
  cats.effect.IO.unsafeRunSync(IO.scala:239)
  io.shiftleft.console.scripting.ScriptManager.runScript(ScriptManager.scala:130)
  io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:64)
  io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:54)
  ammonite.$sess.cmd4$.<clinit>(cmd4.sc:1)

Bye!

Hi Dear Friend,

You can replace all "NodeSteps" with "Traversal", see https://docs.joern.io/upgrade-guides. But after that, here are a new problem that cannot be solved. Have a try, and if you solve it, please tell me. Thanks!

graph-for-funcs.sc:102: type mismatch; found : overflowdb.traversal.Traversal[io.shiftleft.codepropertygraph.generated.nodes.MethodParameterIn] required: ?{def method: ?} Note that implicit conversions are not applicable because they are ambiguous: both method toTrackingPoint in package language of type [A, NodeType <: io.shiftleft.codepropertygraph.generated.nodes.TrackingPointBase](a: A)(implicit f: A => overflowdb.traversal.Traversal[NodeType])io.shiftleft.dataflowengineoss.language.TrackingPoint and method toMethodParameter in package language of type [A](a: A)(implicit f: A => overflowdb.traversal.Traversal[io.shiftleft.codepropertygraph.generated.nodes.MethodParameterIn])io.shiftleft.semanticcpg.language.types.structure.MethodParameter are possible conversion functions from overflowdb.traversal.Traversal[io.shiftleft.codepropertygraph.generated.nodes.MethodParameterIn] to ?{def method: ?} case trackingPoint @ (_: MethodParameterIn) => trackingPoint.to(Traversal).method.head ^ java.lang.RuntimeException: Compilation Failed io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:50) cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:229) cats.effect.internals.IORunLoop$.step(IORunLoop.scala:229) cats.effect.IO.unsafeRunTimed(IO.scala:320) cats.effect.IO.unsafeRunSync(IO.scala:239) io.shiftleft.console.scripting.ScriptManager.runScript(ScriptManager.scala:130) io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:64) io.shiftleft.console.scripting.ScriptManager$CpgScriptRunner.runScript(ScriptManager.scala:54) ammonite.$sess.cmd1$.(cmd1.sc:1)

[zhaohan-xi]

Thanks for your response! Since the Joern is updating week by week (even day by day), I use your old version of graph-for-funcs.sc and also find an old version of Joern (v1.0.111 in its releasing history), now it works!

Also, I want to try your current version of graph-for-funcs.sc, thus could you tell me an approximate time (Month/Year) that when do you make the updating with Joern? I can go to its releasing history to find the version in that time, and that is expected to match the current graph-for-funcs.sc. Thanks!

[GoneZ5]

Thanks for your response! Since the Joern is updating week by week (even day by day), I use your old version of graph-for-funcs.sc and also find an old version of Joern (v1.0.111 in its releasing history), now it works!

Also, I want to try your current version of graph-for-funcs.sc, thus could you tell me an approximate time (Month/Year) that when do you make the updating with Joern? I can go to its releasing history to find the version in that time, and that is expected to match the current graph-for-funcs.sc. Thanks!

Thanks for your response! I install Joern in December 18th, here is my current graph-for-funcs. graph-for-fun.txt

By the way, if I want to use all samples, which files should I modify? I remove the line of 36 in main.py, and decrease the slice_size, but it is frozen in Create task. Could you resolve it?

[zhaohan-xi]

@GoneZ5 When I use the old version of Joern (v1.0.111) + graph-for-func.sc, I touch nothing about these scripts, but set slice_size larger. I found that, when the number of slices is >3, the Create task will freeze and no longer proceed. Thus, in main.py --> select() function --> line 36 "result = result.head(number)", I set that "number" no larger than 3 times of slice_size. For example, if we want to use first N samples, I will set line 36 as result = result.head(N) and slice_size as N/3 (calculate a ceil integer). My test is running on Linux system and not sure whether this is also deployable on your OS.

By the way, the slice_size cannot be too large (at least should <5000, I didn't test the exact value), thus we may never use all samples.

[GoneZ5]

@GoneZ5 When I use the old version of Joern (v1.0.111) + graph-for-func.sc, I touch nothing about these scripts, but set slice_size larger. I found that, when the number of slices is >3, the Create task will freeze and no longer proceed. Thus, in main.py --> select() function --> line 36 "result = result.head(number)", I set that "number" no larger than 3 times of slice_size. For example, if we want to use first N samples, I will set line 36 as result = result.head(N) and slice_size as N/3 (calculate a ceil integer). My test is running on Linux system and not sure whether this is also deployable on your OS.

By the way, the slice_size cannot be too large (at least should <5000, I didn't test the exact value), thus we may never use all samples.

Thank you very much! I try to use your way, it works!

[xshub]

@GoneZ5 When I use the old version of Joern (v1.0.111) + graph-for-func.sc, I touch nothing about these scripts, but set slice_size larger. I found that, when the number of slices is >3, the Create task will freeze and no longer proceed. Thus, in main.py --> select() function --> line 36 "result = result.head(number)", I set that "number" no larger than 3 times of slice_size. For example, if we want to use first N samples, I will set line 36 as result = result.head(N) and slice_size as N/3 (calculate a ceil integer). My test is running on Linux system and not sure whether this is also deployable on your OS.

By the way, the slice_size cannot be too large (at least should <5000, I didn't test the exact value), thus we may never use all samples.

Hi, Dear Friend,

@HarrialX your way, it works! I have another question is that why the number of slices < 3 ? and how to get the problem? I run a python script (a variation of "joern_create" in cpg_generator.py ) just to deal with a single cpg.bin , but I get a error:

`import os import subprocess out_path = 'data/cpg/' in_path = 'data/cpg/' json_file_name = 'my.json' cpg_file='0_cpg.bin' joern_path = 'joern/joern-cli/'

joern_process = subprocess.Popen(["./" + joern_path + "joern"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

json_out = f"{os.path.abspath(out_path)}/{json_file_name}" import_cpg_cmd = f"importCpg(\"{os.path.abspath(in_path)}/{cpg_file}\") \r".encode() script_path = f"{os.path.dirname(os.path.abspath(joern_path))}/graph-for-funcs.sc" run_script_cmd = f"cpg.runScript(\"{script_path}\").toString() |> \"{json_out}\" \r".encode()

joern_process.stdin.write(import_cpg_cmd) print(joern_process.stdout.readline().decode()) joern_process.stdin.write(run_script_cmd) print(joern_process.stdout.readline().decode()) joern_process.stdin.write("delete\r".encode()) print(joern_process.stdout.readline().decode())`

but I got the error:

/bin/stty: 'standard input': Input/output error java.lang.RuntimeException: Nonzero exit value: 1 scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:155) scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:112) ammonite.terminal.TTY$.stty(Utils.scala:114) ammonite.terminal.TTY$.init(Utils.scala:100) ammonite.terminal.Terminal$.readLine(Terminal.scala:41) ammonite.repl.AmmoniteFrontEnd.readLine(AmmoniteFrontEnd.scala:131) ammonite.repl.AmmoniteFrontEnd.action(AmmoniteFrontEnd.scala:25) ammonite.repl.Repl.$anonfun$action$2(Repl.scala:184) ammonite.util.Catching.flatMap(Res.scala:115) ammonite.repl.Repl.action(Repl.scala:168) ammonite.repl.Repl.loop$1(Repl.scala:208) ammonite.repl.Repl.run(Repl.scala:223) ammonite.Main.$anonfun$run$1(Main.scala:224) scala.Option.getOrElse(Option.scala:202) ammonite.Main.run(Main.scala:212) io.shiftleft.console.BridgeBase.runAmmonite(BridgeBase.scala:86) io.shiftleft.console.BridgeBase.runAmmonite$(BridgeBase.scala:60) io.shiftleft.joern.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:5) io.shiftleft.joern.console.AmmoniteBridge$.delayedEndpoint$io$shiftleft$joern$console$AmmoniteBridge$1(AmmoniteBridge.scala:7) io.shiftleft.joern.console.AmmoniteBridge$delayedInit$body.apply(AmmoniteBridge.scala:5) scala.Function0.apply$mcV$sp(Function0.scala:39) scala.Function0.apply$mcV$sp$(Function0.scala:39) scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17) scala.App.$anonfun$main$1(App.scala:75) scala.App.$anonfun$main$1$adapted(App.scala:75) scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576) scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574) scala.collection.AbstractIterable.foreach(Iterable.scala:904) scala.App.main(App.scala:75) scala.App.main$(App.scala:73) io.shiftleft.joern.console.AmmoniteBridge$.main(AmmoniteBridge.scala:5) io.shiftleft.joern.console.AmmoniteBridge.main(AmmoniteBridge.scala)

[xshub]

@GoneZ5 When I use the old version of Joern (v1.0.111) + graph-for-func.sc, I touch nothing about these scripts, but set slice_size larger. I found that, when the number of slices is >3, the Create task will freeze and no longer proceed. Thus, in main.py --> select() function --> line 36 "result = result.head(number)", I set that "number" no larger than 3 times of slice_size. For example, if we want to use first N samples, I will set line 36 as result = result.head(N) and slice_size as N/3 (calculate a ceil integer). My test is running on Linux system and not sure whether this is also deployable on your OS.

By the way, the slice_size cannot be too large (at least should <5000, I didn't test the exact value), thus we may never use all samples.

Hi, Dear Friend,

@HarrialX your way, it works! And I find the reason for Joern freezing maybe is that the "subprocess" package can't deal with well the interactive buffer, then it causes the system to deadlock. there an alternative way is using "pexpect" package to replace the subprocess package.

I using the below code replace the "joern_create" function in cpg_generator.py. And it is useful in my environment.

import pexpect
import os
def joern_create(joern_path, in_path, out_path, cpg_files):
    #print("run in joern_create")
    joern_process = pexpect.spawnu('/bin/bash', ["./" + joern_path + "joern"])
    json_files = []
    for cpg_file in cpg_files:
        json_file_name = f"{cpg_file.split('.')[0]}.json"
        json_files.append(json_file_name)

        print(in_path+cpg_file)
        if os.path.exists(in_path+cpg_file):
            json_out = f"{os.path.abspath(out_path)}/{json_file_name}"
            import_cpg_cmd = f"importCpg(\"{os.path.abspath(in_path)}/{cpg_file}\")"
            script_path = f"{os.path.dirname(os.path.abspath(joern_path))}/graph-for-funcs.sc"
            run_script_cmd = f"cpg.runScript(\"{script_path}\").toString() |> \"{json_out}\" "
            delete_cm = "delete"

            for cmd in [import_cpg_cmd, run_script_cmd, delete_cm]:
                joern_process.expect('joern>', searchwindowsize=50)
                joern_process.sendline(cmd)
                joern_process.buffer = ""
    return json_files

[zhaohan-xi]

@GoneZ5 When I use the old version of Joern (v1.0.111) + graph-for-func.sc, I touch nothing about these scripts, but set slice_size larger. I found that, when the number of slices is >3, the Create task will freeze and no longer proceed. Thus, in main.py --> select() function --> line 36 "result = result.head(number)", I set that "number" no larger than 3 times of slice_size. For example, if we want to use first N samples, I will set line 36 as result = result.head(N) and slice_size as N/3 (calculate a ceil integer). My test is running on Linux system and not sure whether this is also deployable on your OS. By the way, the slice_size cannot be too large (at least should <5000, I didn't test the exact value), thus we may never use all samples.

Hi, Dear Friend,

@HarrialX your way, it works! And I find the reason for Joern freezing maybe is that the "subprocess" package can't deal with well the interactive buffer, then it causes the system to deadlock. there an alternative way is using "pexpect" package to replace the subprocess package.

I using the below code replace the "joern_create" function in cpg_generator.py. And it is useful in my environment.

import pexpect
import os
def joern_create(joern_path, in_path, out_path, cpg_files):
    #print("run in joern_create")
    joern_process = pexpect.spawnu('/bin/bash', ["./" + joern_path + "joern"])
    json_files = []
    for cpg_file in cpg_files:
        json_file_name = f"{cpg_file.split('.')[0]}.json"
        json_files.append(json_file_name)

        print(in_path+cpg_file)
        if os.path.exists(in_path+cpg_file):
            json_out = f"{os.path.abspath(out_path)}/{json_file_name}"
            import_cpg_cmd = f"importCpg(\"{os.path.abspath(in_path)}/{cpg_file}\")"
            script_path = f"{os.path.dirname(os.path.abspath(joern_path))}/graph-for-funcs.sc"
            run_script_cmd = f"cpg.runScript(\"{script_path}\").toString() |> \"{json_out}\" "
            delete_cm = "delete"

            for cmd in [import_cpg_cmd, run_script_cmd, delete_cm]:
                joern_process.expect('joern>', searchwindowsize=50)
                joern_process.sendline(cmd)
                joern_process.buffer = ""
    return json_files

Hi @xshub, I test and find that only slices num <=3, it can keep proceeding with the raw codes, and not sure how it comes in sysem.

[nimasteryang]

Hi @epicosy , I have modified the scala script a little bit and now it works with the latest version of Joern. Do you want me to submit a commit?

[itismyname]

Hi @epicosy , I have modified the scala script a little bit and now it works with the latest version of Joern. Do you want me to submit a commit?

Hi @nimasteryang , Would you mind sending me one? It would be great helpful.

[nimasteryang]

Hi @epicosy , I have modified the scala script a little bit and now it works with the latest version of Joern. Do you want me to submit a commit?

Hi @nimasteryang , Would you mind sending me one? It would be great helpful.

Hi, here is the script, this script is tested and work with Joern V1.1.172 (use --interactive to select specific install version when install joern). One problem is sometime this script can't give PDG. graph-for-funcs.txt

[itismyname]

Hi @epicosy , I have modified the scala script a little bit and now it works with the latest version of Joern. Do you want me to submit a commit?

Hi @nimasteryang , Would you mind sending me one? It would be great helpful.

Hi, here is the script, this script is tested and work with Joern V1.1.172 (use --interactive to select specific install version when install joern). One problem is sometime this script can't give PDG. graph-for-funcs.txt

It's work! Thank a lot .

[lxyeternal]

Compiling /home/gwb/VulDetect/devign/joern/graph-for-funcs.sc graph-for-funcs.sc:29: object CfgNode is not a member of package io.shiftleft.semanticcpg.language.types.expressions.generalizations import io.shiftleft.semanticcpg.language.types.expressions.generalizations.CfgNode ^ graph-for-funcs.sc:35: object Call is not a member of package io.shiftleft.semanticcpg.language.types.expressions import io.shiftleft.semanticcpg.language.types.expressions.Call ^ graph-for-funcs.sc:36: object Local is not a member of package io.shiftleft.semanticcpg.language.types.structure import io.shiftleft.semanticcpg.language.types.structure.Local ^ graph-for-funcs.sc:50: not found: type OdbEdge implicit val encodeEdge: Encoder[OdbEdge] = ^ graph-for-funcs.sc:51: not found: type OdbEdge (edge: OdbEdge) => ^ graph-for-funcs.sc:63: could not find implicit value for parameter encoder: io.circe.Encoder[overflowdb.Edge] Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))), ^ graph-for-funcs.sc:81: not found: type Vertex val methodVertex: Vertex = method //TODO MP drop as soon as we have the remainder of the below in ODB graph api ^ java.lang.RuntimeException: Compilation Failed io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:52) map @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:49) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$5(AmmoniteExecutor.scala:45) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$3(AmmoniteExecutor.scala:44) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.runScript(AmmoniteExecutor.scala:43)

Hello, I ran this code and reported the above error, and I don’t know how to solve it.

[mnixx]

Compiling /home/gwb/VulDetect/devign/joern/graph-for-funcs.sc graph-for-funcs.sc:29: object CfgNode is not a member of package io.shiftleft.semanticcpg.language.types.expressions.generalizations import io.shiftleft.semanticcpg.language.types.expressions.generalizations.CfgNode ^ graph-for-funcs.sc:35: object Call is not a member of package io.shiftleft.semanticcpg.language.types.expressions import io.shiftleft.semanticcpg.language.types.expressions.Call ^ graph-for-funcs.sc:36: object Local is not a member of package io.shiftleft.semanticcpg.language.types.structure import io.shiftleft.semanticcpg.language.types.structure.Local ^ graph-for-funcs.sc:50: not found: type OdbEdge implicit val encodeEdge: Encoder[OdbEdge] = ^ graph-for-funcs.sc:51: not found: type OdbEdge (edge: OdbEdge) => ^ graph-for-funcs.sc:63: could not find implicit value for parameter encoder: io.circe.Encoder[overflowdb.Edge] Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))), ^ graph-for-funcs.sc:81: not found: type Vertex val methodVertex: Vertex = method //TODO MP drop as soon as we have the remainder of the below in ODB graph api ^ java.lang.RuntimeException: Compilation Failed io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:52) map @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:49) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$5(AmmoniteExecutor.scala:45) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$3(AmmoniteExecutor.scala:44) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.runScript(AmmoniteExecutor.scala:43)

Hello, I ran this code and reported the above error, and I don’t know how to solve it.

I've meet the same problem as yours, thus I use the solution provided by @nimasteryang , then I met a lot of WARN , but finally it works. the WARN just like these ⬇️

[little-little-fox]

Compiling /home/gwb/VulDetect/devign/joern/graph-for-funcs.sc graph-for-funcs.sc:29: object CfgNode is not a member of package io.shiftleft.semanticcpg.language.types.expressions.generalizations import io.shiftleft.semanticcpg.language.types.expressions.generalizations.CfgNode ^ graph-for-funcs.sc:35: object Call is not a member of package io.shiftleft.semanticcpg.language.types.expressions import io.shiftleft.semanticcpg.language.types.expressions.Call ^ graph-for-funcs.sc:36: object Local is not a member of package io.shiftleft.semanticcpg.language.types.structure import io.shiftleft.semanticcpg.language.types.structure.Local ^ graph-for-funcs.sc:50: not found: type OdbEdge implicit val encodeEdge: Encoder[OdbEdge] = ^ graph-for-funcs.sc:51: not found: type OdbEdge (edge: OdbEdge) => ^ graph-for-funcs.sc:63: could not find implicit value for parameter encoder: io.circe.Encoder[overflowdb.Edge] Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))), ^ graph-for-funcs.sc:81: not found: type Vertex val methodVertex: Vertex = method //TODO MP drop as soon as we have the remainder of the below in ODB graph api ^ java.lang.RuntimeException: Compilation Failed io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:52) map @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:49) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$5(AmmoniteExecutor.scala:45) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$3(AmmoniteExecutor.scala:44) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.runScript(AmmoniteExecutor.scala:43)

Hello, I ran this code and reported the above error, and I don’t know how to solve it.

I meet the same errors with you! I want to konw if you have resolved this problem.

[lxyeternal]

Compiling /home/gwb/VulDetect/devign/joern/graph-for-funcs.sc graph-for-funcs.sc:29: object CfgNode is not a member of package io.shiftleft.semanticcpg.language.types.expressions.generalizations import io.shiftleft.semanticcpg.language.types.expressions.generalizations.CfgNode ^ graph-for-funcs.sc:35: object Call is not a member of package io.shiftleft.semanticcpg.language.types.expressions import io.shiftleft.semanticcpg.language.types.expressions.Call ^ graph-for-funcs.sc:36: object Local is not a member of package io.shiftleft.semanticcpg.language.types.structure import io.shiftleft.semanticcpg.language.types.structure.Local ^ graph-for-funcs.sc:50: not found: type OdbEdge implicit val encodeEdge: Encoder[OdbEdge] = ^ graph-for-funcs.sc:51: not found: type OdbEdge (edge: OdbEdge) => ^ graph-for-funcs.sc:63: could not find implicit value for parameter encoder: io.circe.Encoder[overflowdb.Edge] Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))), ^ graph-for-funcs.sc:81: not found: type Vertex val methodVertex: Vertex = method //TODO MP drop as soon as we have the remainder of the below in ODB graph api ^ java.lang.RuntimeException: Compilation Failed io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:52) map @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:49) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$5(AmmoniteExecutor.scala:45) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$3(AmmoniteExecutor.scala:44) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.runScript(AmmoniteExecutor.scala:43) Hello, I ran this code and reported the above error, and I don’t know how to solve it.

I've meet the same problem as yours, thus I use the solution provided by @nimasteryang , then I met a lot of WARN , but finally it works. the WARN just like these ⬇️

My error is that there is a problem with the .sc file, indicating that the imported package is incorrect. I don’t know how to solve it. My joern can run normally and generate a .bin file.

[mnixx]

Compiling /home/gwb/VulDetect/devign/joern/graph-for-funcs.sc graph-for-funcs.sc:29: object CfgNode is not a member of package io.shiftleft.semanticcpg.language.types.expressions.generalizations import io.shiftleft.semanticcpg.language.types.expressions.generalizations.CfgNode ^ graph-for-funcs.sc:35: object Call is not a member of package io.shiftleft.semanticcpg.language.types.expressions import io.shiftleft.semanticcpg.language.types.expressions.Call ^ graph-for-funcs.sc:36: object Local is not a member of package io.shiftleft.semanticcpg.language.types.structure import io.shiftleft.semanticcpg.language.types.structure.Local ^ graph-for-funcs.sc:50: not found: type OdbEdge implicit val encodeEdge: Encoder[OdbEdge] = ^ graph-for-funcs.sc:51: not found: type OdbEdge (edge: OdbEdge) => ^ graph-for-funcs.sc:63: could not find implicit value for parameter encoder: io.circe.Encoder[overflowdb.Edge] Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))), ^ graph-for-funcs.sc:81: not found: type Vertex val methodVertex: Vertex = method //TODO MP drop as soon as we have the remainder of the below in ODB graph api ^ java.lang.RuntimeException: Compilation Failed io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:52) map @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:49) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$5(AmmoniteExecutor.scala:45) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.$anonfun$runScript$3(AmmoniteExecutor.scala:44) flatMap @ io.shiftleft.console.scripting.AmmoniteExecutor.runScript(AmmoniteExecutor.scala:43) Hello, I ran this code and reported the above error, and I don’t know how to solve it.

I've meet the same problem as yours, thus I use the solution provided by @nimasteryang , then I met a lot of WARN , but finally it works. the WARN just like these ⬇️

My error is that there is a problem with the .sc file, indicating that the imported package is incorrect. I don’t know how to solve it. My joern can run normally and generate a .bin file.

for -p stage, I just replace the .sc file with the one provided by @nimasteryang, and you can find it in this issues.

[1112z-c]

hi

[txg666]

I had met many different errors when running the graph-for-funcs.sc with different versions of Joern, such as "Mismatch methods...","could not find implicit value for parameter encoder:..." and so on. So, I finally found that the version of Joern with 1.0.170 works fine!

[faysalhossain2007]

Hi @epicosy , I have modified the scala script a little bit and now it works with the latest version of Joern. Do you want me to submit a commit?

Hi @nimasteryang , Would you mind sending me one? It would be great helpful.

Hi, here is the script, this script is tested and work with Joern V1.1.172 (use --interactive to select specific install version when install joern). One problem is sometime this script can't give PDG. graph-for-funcs.txt

I am facing the following issue with your graphs-for-funcs.txt. Do you have any suggestions?

/home/faysal/bin/joern/graph-for-funcs.sc:29: object CfgNode is not a member of package io.shiftleft.semanticcpg.language.types.expressions.generalizations
import io.shiftleft.semanticcpg.language.types.expressions.generalizations.CfgNode
       ^
/home/faysal/bin/joern/graph-for-funcs.sc:33: object dataflowengineoss is not a member of package io.shiftleft
import io.shiftleft.dataflowengineoss.language._
                    ^
/home/faysal/bin/joern/graph-for-funcs.sc:35: object Call is not a member of package io.shiftleft.semanticcpg.language.types.expressions
import io.shiftleft.semanticcpg.language.types.expressions.Call
       ^
/home/faysal/bin/joern/graph-for-funcs.sc:36: object Local is not a member of package io.shiftleft.semanticcpg.language.types.structure
import io.shiftleft.semanticcpg.language.types.structure.Local
       ^
/home/faysal/bin/joern/graph-for-funcs.sc:50: not found: type OdbEdge
implicit val encodeEdge: Encoder[OdbEdge] =
                                 ^
/home/faysal/bin/joern/graph-for-funcs.sc:51: not found: type OdbEdge
  (edge: OdbEdge) =>
         ^
/home/faysal/bin/joern/graph-for-funcs.sc:63: could not find implicit value for parameter encoder: io.circe.Encoder[overflowdb.Edge]
        Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))),
                                                                                      ^
/home/faysal/bin/joern/graph-for-funcs.sc:64: value propertyMap is not a member of io.shiftleft.codepropertygraph.generated.nodes.AstNode
      ("properties", Json.fromValues(node.propertyMap.asScala.toList.map { case (key, value) =>
                                          ^
/home/faysal/bin/joern/graph-for-funcs.sc:66: type mismatch;
 found   : Any
 required: String
          ("key", Json.fromString(key)),
                                  ^
/home/faysal/bin/joern/graph-for-funcs.sc:81: not found: type Vertex
      val methodVertex: Vertex = method //TODO MP drop as soon as we have the remainder of the below in ODB graph api
                        ^
java.lang.RuntimeException: Compilation Failed
  io.joern.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:59)
  io.joern.console.scripting.AmmoniteExecutor$$Lambda$1894/0x0000000000000000.apply(Unknown Source)
  map @ io.joern.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:56)
  apply @ io.joern.console.scripting.AmmoniteExecutor.$anonfun$runScript$5(AmmoniteExecutor.scala:47)
  flatMap @ io.joern.console.scripting.AmmoniteExecutor.$anonfun$runScript$5(AmmoniteExecutor.scala:47)
  flatMap @ io.joern.console.scripting.AmmoniteExecutor.$anonfun$runScript$3(AmmoniteExecutor.scala:46)
  apply @ io.joern.console.scripting.AmmoniteExecutor.runScript(AmmoniteExecutor.scala:45)
  flatMap @ io.joern.console.scripting.AmmoniteExecutor.runScript(AmmoniteExecutor.scala:45)

[Vinothini-G]

Hi @epicosy , I have modified the scala script a little bit and now it works with the latest version of Joern. Do you want me to submit a commit?

Hi @nimasteryang , Would you mind sending me one? It would be great helpful.

Hi, here is the script, this script is tested and work with Joern V1.1.172 (use --interactive to select specific install version when install joern). One problem is sometime this script can't give PDG. graph-for-funcs.txt

I am facing the following issue with your graphs-for-funcs.txt. Do you have any suggestions?

/home/faysal/bin/joern/graph-for-funcs.sc:29: object CfgNode is not a member of package io.shiftleft.semanticcpg.language.types.expressions.generalizations
import io.shiftleft.semanticcpg.language.types.expressions.generalizations.CfgNode
       ^
/home/faysal/bin/joern/graph-for-funcs.sc:33: object dataflowengineoss is not a member of package io.shiftleft
import io.shiftleft.dataflowengineoss.language._
                    ^
/home/faysal/bin/joern/graph-for-funcs.sc:35: object Call is not a member of package io.shiftleft.semanticcpg.language.types.expressions
import io.shiftleft.semanticcpg.language.types.expressions.Call
       ^
/home/faysal/bin/joern/graph-for-funcs.sc:36: object Local is not a member of package io.shiftleft.semanticcpg.language.types.structure
import io.shiftleft.semanticcpg.language.types.structure.Local
       ^
/home/faysal/bin/joern/graph-for-funcs.sc:50: not found: type OdbEdge
implicit val encodeEdge: Encoder[OdbEdge] =
                                 ^
/home/faysal/bin/joern/graph-for-funcs.sc:51: not found: type OdbEdge
  (edge: OdbEdge) =>
         ^
/home/faysal/bin/joern/graph-for-funcs.sc:63: could not find implicit value for parameter encoder: io.circe.Encoder[overflowdb.Edge]
        Json.fromValues((node.inE("AST", "CFG").l ++ node.outE("AST", "CFG").l).map(_.asJson))),
                                                                                      ^
/home/faysal/bin/joern/graph-for-funcs.sc:64: value propertyMap is not a member of io.shiftleft.codepropertygraph.generated.nodes.AstNode
      ("properties", Json.fromValues(node.propertyMap.asScala.toList.map { case (key, value) =>
                                          ^
/home/faysal/bin/joern/graph-for-funcs.sc:66: type mismatch;
 found   : Any
 required: String
          ("key", Json.fromString(key)),
                                  ^
/home/faysal/bin/joern/graph-for-funcs.sc:81: not found: type Vertex
      val methodVertex: Vertex = method //TODO MP drop as soon as we have the remainder of the below in ODB graph api
                        ^
java.lang.RuntimeException: Compilation Failed
  io.joern.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:59)
  io.joern.console.scripting.AmmoniteExecutor$$Lambda$1894/0x0000000000000000.apply(Unknown Source)
  map @ io.joern.console.scripting.AmmoniteExecutor.$anonfun$runScript$7(AmmoniteExecutor.scala:56)
  apply @ io.joern.console.scripting.AmmoniteExecutor.$anonfun$runScript$5(AmmoniteExecutor.scala:47)
  flatMap @ io.joern.console.scripting.AmmoniteExecutor.$anonfun$runScript$5(AmmoniteExecutor.scala:47)
  flatMap @ io.joern.console.scripting.AmmoniteExecutor.$anonfun$runScript$3(AmmoniteExecutor.scala:46)
  apply @ io.joern.console.scripting.AmmoniteExecutor.runScript(AmmoniteExecutor.scala:45)
  flatMap @ io.joern.console.scripting.AmmoniteExecutor.runScript(AmmoniteExecutor.scala:45)

I'm facing the same problem with the latest version. Has anyone resolved it?

[IntHelloWorld]

Hi @epicosy, as Joern is updating every day, the graph-for-funcs.sc is not adjusted for the latest Joern anymore. I have modified the script and now it can be run in the newest Joern. (version 1.1.1456, 2023/2/17) Here is the script: graph-for-funcs.txt

[LovelyBuggies]

Hi @epicosy, as Joern is updating every day, the graph-for-funcs.sc is not adjusted for the latest Joern anymore. I have modified the script and now it can be run in the newest Joern. (version 1.1.1456, 2023/2/17) Here is the script: graph-for-funcs.txt

Thank you @IntHelloWorld!. Now, I'm using Joern 1.1.1449, after replacing with your scripts, I have 0_cpg.bin and 0_cpg.json after creating the task. But I get the following errors:

joern> Would you like to save changes? (y/N)

Traceback (most recent call last):
  File "main.py", line 157, in <module>
    main()
  File "main.py", line 146, in main
    create_task()
  File "main.py", line 61, in create_task
    graphs = prepare.json_process(PATHS.cpg, json_file)
  File "/Users/ninolau/Documents/GitHub/devign/src/prepare/cpg_generator.py", line 74, in json_process
    container = [graph_indexing(graph) for graph in cpg_json["functions"] if graph["file"] != "N/A"]
  File "/Users/ninolau/Documents/GitHub/devign/src/prepare/cpg_generator.py", line 74, in <listcomp>
    container = [graph_indexing(graph) for graph in cpg_json["functions"] if graph["file"] != "N/A"]
  File "/Users/ninolau/Documents/GitHub/devign/src/prepare/cpg_generator.py", line 24, in graph_indexing
    idx = int(graph["file"].split(".c")[0].split("/")[-1])
ValueError: invalid literal for int() with base 10: '<empty>'

I checked 0_cpg.json and I found some in AST properties field. Do I create the CPGs successfully? If not, how should I solve it?

[IntHelloWorld]

Hi @epicosy, as Joern is updating every day, the graph-for-funcs.sc is not adjusted for the latest Joern anymore. I have modified the script and now it can be run in the newest Joern. (version 1.1.1456, 2023/2/17) Here is the script: graph-for-funcs.txt

Thank you @IntHelloWorld!. Now, I'm using Joern 1.1.1449, after replacing with your scripts, I have 0_cpg.bin and 0_cpg.json after creating the task. But I get the following errors:

joern> Would you like to save changes? (y/N)

Traceback (most recent call last):
  File "main.py", line 157, in <module>
    main()
  File "main.py", line 146, in main
    create_task()
  File "main.py", line 61, in create_task
    graphs = prepare.json_process(PATHS.cpg, json_file)
  File "/Users/ninolau/Documents/GitHub/devign/src/prepare/cpg_generator.py", line 74, in json_process
    container = [graph_indexing(graph) for graph in cpg_json["functions"] if graph["file"] != "N/A"]
  File "/Users/ninolau/Documents/GitHub/devign/src/prepare/cpg_generator.py", line 74, in <listcomp>
    container = [graph_indexing(graph) for graph in cpg_json["functions"] if graph["file"] != "N/A"]
  File "/Users/ninolau/Documents/GitHub/devign/src/prepare/cpg_generator.py", line 24, in graph_indexing
    idx = int(graph["file"].split(".c")[0].split("/")[-1])
ValueError: invalid literal for int() with base 10: '<empty>'

I checked 0_cpg.json and I found some in AST properties field. Do I create the CPGs successfully? If not, how should I solve it?

In the new version of Joern, the value of graph["file"] when the graph belongs to no file is "<empty>" rather than "N/A", so simply replace "N/A" with "<empty>" will be ok :)