Be able to pass frontend args to joern console

[a0029328]

I need to run a script, since it can only be executed via BridgeBase, I can only use the “joern” script directly, not joern-parse, and so on. I also need the analyzer to ignore test directories when scanning. Is it possible to specify this in command line arguments or in the script itself?

For example, i want to be able to do that:

joern . --language python --frontend-args --exclude-regex="test*" --script /path/to/script.sc --param mycustomparam='value'

On the latest version of joern i get this exceptions:

Warning: Unknown option --frontend-args
Warning: Unknown option --exclude-regex=test*

Regards

[maltek]

I assume you use importCode in your script? You can specify extra frontend args there, e.g. importCode.python("<path>", args=List("--foo", "--bar"))

[a0029328]

@maltek Sorry, that did not work

I'm using importCode (there can be multiple languages)

Getting this exception

  |method apply in class ImportCode: (inputPath: String, projectName: String, language: String):
  |  io.shiftleft.codepropertygraph.generated.Cpg does not have a parameter args

Even if i am setting language like importCode.python("<path>", args=List("--exclude-regex", "test*")), there is also an error:

Error: Unknown option --exclude-regex
Error: Unknown argument 'test*'
Usage:  [options]

java.lang.RuntimeException: unable to parse XTypeRecoveryConfig from commandline arguments --exclude-regex test*
  at io.joern.x2cpg.passes.frontend.XTypeRecoveryConfig$.parse$$anonfun$1(XTypeRecovery.scala:44)
  at scala.Option.getOrElse(Option.scala:201)
  at io.joern.x2cpg.passes.frontend.XTypeRecoveryConfig$.parse(XTypeRecovery.scala:45)
  at io.joern.console.cpgcreation.PythonSrcCpgGenerator.generate(PythonSrcCpgGenerator.scala:23)
  at io.joern.console.cpgcreation.CpgGeneratorFactory.runGenerator(CpgGeneratorFactory.scala:59)
  at io.joern.console.cpgcreation.ImportCode.$anonfun$7(ImportCode.scala:235)
  at scala.Option.flatMap(Option.scala:283)
  at io.joern.console.cpgcreation.ImportCode.io$joern$console$cpgcreation$ImportCode$$apply(ImportCode.scala:240)
  at io.joern.console.cpgcreation.ImportCode$Frontend.apply(ImportCode.scala:123)

[maltek]

@mpollmeier that error seems to be a regression from https://github.com/joernio/joern/pull/4684 (the frontend args are also getting passed to the type recovery which can't deal with them)

[mpollmeier]

The parameter handling for additional unbounded args unfortunately deviates from standards. Typically in a case like this one would use -- as a delimitor.

I played around with different variants and the easiest and cleanest solution seems to me to ignore all unknown/unwanted arguments when calling XTypeRecoveryConfig.parse. We could configure the scopt parser to not error on those, but that would still print warnings - that would be https://github.com/joernio/joern/pull/4900

Instead we probably need to filter out everything but the expected parameters... What do you think @maltek?

[mpollmeier]

@a0029328 we just merged a fix, can you please check if this works for you?

[a0029328]

@mpollmeier,

Yep, thanks for your assistance. The --exclude argument is functioning correctly when I specify the absolute path to the directory, resulting in fewer nodes in the output (without files in specified dir). However, I am experiencing issues with the --exclude-regex option. When I specify the pattern test*.py, I expect that Python files with names starting with test will not be scanned. But these files still appear in the resulting CPG. Could you please investigate this on your end? It’s possible that I may not be using it correctly

Thank you!