search

AppThreat / atom

Atom is a novel intermediate representation for applications and a standalone tool that is powered by chen.
https://appthreat.com
Apache License 2.0
52 stars 2 forks source link

Exception in thread "main" overflowdb.traversal.filter.StringPropertyFilter$InvalidRegexException: invalid regular expression #154

Open almaz045 opened 3 months ago

almaz045 commented 3 months ago

I've tried to test depscan on https://github.com/grpc/grpc by next command:

atom reachables -o app.atom -s reachables.json -l c .

p.s. bom.json is exist

And got next error:

Generating data-flow dependencies from atom. Please wait ...
Exception in thread "main" overflowdb.traversal.filter.StringPropertyFilter$InvalidRegexException: invalid regular expression: ` grpc_core.ChildPolicyHandler.ConfigChangeRequiresNewPolicyInstance:bool(grpc_core::LoadBalancingPolicy::Config *`
    at overflowdb.traversal.filter.StringPropertyFilter$.regexpCompile(StringPropertyFilter.scala:48)
    at overflowdb.traversal.filter.StringPropertyFilter$.regexp(StringPropertyFilter.scala:13)
    at io.shiftleft.codepropertygraph.generated.traversal.CallTraversalExtGen$.typeFullName$extension(Call.scala:597)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5$$anonfun$2$$anonfun$5(CdxPass.scala:285)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5$$anonfun$2(CdxPass.scala:351)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.immutable.Vector.foreach(Vector.scala:2124)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5(CdxPass.scala:353)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.immutable.VectorStatics$.foreachRec(Vector.scala:2124)
    at scala.collection.immutable.Vector.foreach(Vector.scala:2130)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1(CdxPass.scala:354)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
    at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1300)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run(CdxPass.scala:355)
    at io.shiftleft.passes.CpgPass.runOnPart(CpgPass.scala:29)
    at io.shiftleft.passes.NewStyleCpgPassBase.runWithBuilder(CpgPass.scala:134)
    at io.shiftleft.passes.ForkJoinParallelCpgPass.createApplySerializeAndStore(CpgPass.scala:77)
    at io.shiftleft.passes.NewStyleCpgPassBase.createAndApply(CpgPass.scala:124)
    at io.appthreat.atom.Atom$.generateForLanguage(Atom.scala:606)
    at io.appthreat.atom.Atom$.generateAtom(Atom.scala:446)
    at io.appthreat.atom.Atom$.run(Atom.scala:300)
    at io.appthreat.atom.Atom$.run(Atom.scala:294)
    at io.appthreat.atom.Atom$.main(Atom.scala:275)
    at io.appthreat.atom.Atom.main(Atom.scala)
Caused by: java.util.regex.PatternSyntaxException: Unclosed group near index 117
(?s) grpc_core.ChildPolicyHandler.ConfigChangeRequiresNewPolicyInstance:bool(grpc_core::LoadBalancingPolicy::Config *
    at java.base/java.util.regex.Pattern.error(Pattern.java:2204)
    at java.base/java.util.regex.Pattern.accept(Pattern.java:2054)
    at java.base/java.util.regex.Pattern.group0(Pattern.java:3232)
    at java.base/java.util.regex.Pattern.sequence(Pattern.java:2300)
    at java.base/java.util.regex.Pattern.expr(Pattern.java:2245)
    at java.base/java.util.regex.Pattern.compile(Pattern.java:1945)
    at java.base/java.util.regex.Pattern.<init>(Pattern.java:1576)
    at java.base/java.util.regex.Pattern.compile(Pattern.java:1101)
    at scala.util.matching.Regex.<init>(Regex.scala:234)
    at scala.collection.StringOps$.r$extension(StringOps.scala:852)
    at overflowdb.traversal.filter.StringPropertyFilter$.regexpCompile(StringPropertyFilter.scala:45)
    ... 32 more
prabhu commented 3 months ago

Interesting test case! There is a type that is confusing the regex during tagging.

prabhu commented 2 months ago

@almaz045 could you confirm the version of atom used, since I don't get this error with the latest.

kittyandrew commented 1 month ago

Hey, I've got very similar crash on ghcr.io/appthreat/atom:nightly (as of today) on my repo

Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
Generating data-flow dependencies from atom. Please wait ...
Exception in thread "main" overflowdb.traversal.filter.StringPropertyFilter$InvalidRegexException: invalid regular expression: `,_a).py:<module>.*`
    at overflowdb.traversal.filter.StringPropertyFilter$.regexpCompile(StringPropertyFilter.scala:48)
    at overflowdb.traversal.filter.StringPropertyFilter$.regexp(StringPropertyFilter.scala:13)
    at io.shiftleft.codepropertygraph.generated.traversal.CallTraversalExtGen$.methodFullName$extension(Call.scala:361)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5$$anonfun$1$$anonfun$1$$anonfun$1(CdxPass.scala:118)
    at overflowdb.traversal.TraversalLogicExt$.where$extension$$anonfun$1(Traversal.scala:176)
    at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:479)
    at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:583)
    at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
    at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1303)
    at io.shiftleft.semanticcpg.language.NewTagNodePairTraversal.store(NewTagNodePairTraversal.scala:9)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5$$anonfun$1$$anonfun$1(CdxPass.scala:119)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.immutable.Set$Set2.foreach(Set.scala:210)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5$$anonfun$1(CdxPass.scala:114)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.immutable.Set$Set1.foreach(Set.scala:177)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5(CdxPass.scala:112)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.immutable.Vector.foreach(Vector.scala:2124)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1(CdxPass.scala:91)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
    at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
    at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
    at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1303)
    at io.appthreat.x2cpg.passes.taggers.CdxPass.run(CdxPass.scala:64)
    at io.shiftleft.passes.CpgPass.runOnPart(CpgPass.scala:29)
    at io.shiftleft.passes.NewStyleCpgPassBase.runWithBuilder(CpgPass.scala:134)
    at io.shiftleft.passes.ForkJoinParallelCpgPass.createApplySerializeAndStore(CpgPass.scala:77)
    at io.shiftleft.passes.NewStyleCpgPassBase.createAndApply(CpgPass.scala:124)
    at io.appthreat.atom.Atom$.generateForLanguage(Atom.scala:606)
    at io.appthreat.atom.Atom$.generateAtom(Atom.scala:446)
    at io.appthreat.atom.Atom$.run(Atom.scala:300)
    at io.appthreat.atom.Atom$.run(Atom.scala:294)
    at io.appthreat.atom.Atom$.main(Atom.scala:275)
    at io.appthreat.atom.Atom.main(Atom.scala)
Caused by: java.util.regex.PatternSyntaxException: Unmatched closing ')' near index 6
(?s),_a).py:<module>.*
      ^
    at java.base/java.util.regex.Pattern.error(Pattern.java:2204)
    at java.base/java.util.regex.Pattern.compile(Pattern.java:1949)
    at java.base/java.util.regex.Pattern.<init>(Pattern.java:1576)
    at java.base/java.util.regex.Pattern.compile(Pattern.java:1101)
    at scala.util.matching.Regex.<init>(Regex.scala:234)
    at scala.collection.StringOps$.r$extension(StringOps.scala:852)
    at overflowdb.traversal.filter.StringPropertyFilter$.regexpCompile(StringPropertyFilter.scala:45)
    ... 39 more
prabhu commented 1 month ago

@kittyandrew, could you share the .py file or the code snippet?

kittyandrew commented 1 month ago

Hi, I've looked at it again today and realized that I might've included .venv directory when pointing to the "current working directory". Problem went away when I started pointing only to my source code.

I don't know if that makes sense to debug further

prabhu commented 1 month ago

Thank you for confirming. Will keep this issue open in case we face any other problematic pattern in the future.