search

microsoft / ApplicationInspector

A source code analyzer built for surfacing features of interest and other characteristics to answer the question 'What's in the code?' quickly using static analysis with a json based rules engine. Ideal for scanning components before use or detecting feature level changes.
MIT License
4.26k stars 356 forks source link

Stack overflow exception in TextContainer.GetPrefixLocation #583

Open jhutchings1 opened 7 months ago

jhutchings1 commented 7 months ago

Describe the bug I ran ApplicationInspector against the dependabot-core repo, and it crashed with a stackoverflow exception.

To Reproduce Steps to reproduce the behavior:

  1. git clone https://github.com/dependabot/dependabot-core
  2. appinspector analyze -s dependabot-core -N

Expected behavior Application inspector runs and outputs the analysis

Operating Environment (please complete the following information):

Additional context Here's a snippet of the log before the crash (there are 40k instances of this first line before it dies)

   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.GetPrefixLocation(Int32, Int32, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.PopulateCommentedStatesInternal(Int32, System.String, System.String, Boolean)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.PopulateCommentedState(Int32)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.IsCommented(Int32)
   at Microsoft.ApplicationInspector.RulesEngine.TextContainer.ScopeMatch(System.Collections.Generic.IList`1<Microsoft.ApplicationInspector.RulesEngine.PatternScope>, Microsoft.ApplicationInspector.RulesEngine.Boundary)
   at Microsoft.ApplicationInspector.RulesEngine.OatExtensions.OatSubstringIndexOperation+<GetMatches>d__4.MoveNext()
   at System.Linq.Enumerable+SelectEnumerableIterator`2[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.ValueTuple`2[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Collections.Generic.List`1[[System.ValueTuple`2[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].AddRange(System.Collections.Generic.IEnumerable`1<System.ValueTuple`2<Int32,System.__Canon>>)
   at Microsoft.ApplicationInspector.RulesEngine.OatExtensions.OatSubstringIndexOperation.SubstringIndexOperationDelegate(Microsoft.CST.OAT.Clause, System.Object, System.Object, System.Collections.Generic.IEnumerable`1<Microsoft.CST.OAT.ClauseCapture>)
   at Microsoft.CST.OAT.Analyzer.GetClauseCapture(Microsoft.CST.OAT.Clause, System.Object, System.Object, System.Collections.Generic.IEnumerable`1<Microsoft.CST.OAT.ClauseCapture>)
   at Microsoft.CST.OAT.Analyzer.Evaluate(System.String[], System.Collections.Generic.List`1<Microsoft.CST.OAT.Clause>, System.Object, System.Object, System.Collections.Generic.IEnumerable`1<Microsoft.CST.OAT.ClauseCapture>)
   at Microsoft.CST.OAT.Analyzer+<>c__DisplayClass34_1.<Evaluate>g__EvaluateLambda|1()
   at Microsoft.CST.OAT.Analyzer.Evaluate(System.String[], System.Collections.Generic.List`1<Microsoft.CST.OAT.Clause>, System.Object, System.Object, System.Collections.Generic.IEnumerable`1<Microsoft.CST.OAT.ClauseCapture>)
   at Microsoft.CST.OAT.Analyzer.GetCapture(Microsoft.CST.OAT.Rule, System.Object, System.Object)
   at Microsoft.CST.OAT.Analyzer+<>c__DisplayClass22_0.<GetCaptures>b__0(Microsoft.CST.OAT.Rule)
   at System.Threading.Tasks.Parallel+<>c__DisplayClass43_0`2[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<PartitionerForEachWorker>b__1(System.Collections.IEnumerator ByRef, Int64, Boolean ByRef)
   at System.Threading.Tasks.TaskReplicator+Replica.Execute()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean)
   at System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler, Boolean)
   at System.Threading.Tasks.TaskReplicator.Run[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](ReplicatableUserAction`1<System.__Canon>, System.Threading.Tasks.ParallelOptions, Boolean)
   at System.Threading.Tasks.Parallel.PartitionerForEachWorker[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Concurrent.Partitioner`1<System.__Canon>, System.Threading.Tasks.ParallelOptions, System.Action`1<System.__Canon>, System.Action`2<System.__Canon,System.Threading.Tasks.ParallelLoopState>, System.Action`3<System.__Canon,System.Threading.Tasks.ParallelLoopState,Int64>, System.Func`4<System.__Canon,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`5<System.__Canon,System.Threading.Tasks.ParallelLoopState,Int64,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>)
   at System.Threading.Tasks.Parallel.ForEachWorker[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Threading.Tasks.ParallelOptions, System.Action`1<System.__Canon>, System.Action`2<System.__Canon,System.Threading.Tasks.ParallelLoopState>, System.Action`3<System.__Canon,System.Threading.Tasks.ParallelLoopState,Int64>, System.Func`4<System.__Canon,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`5<System.__Canon,System.Threading.Tasks.ParallelLoopState,Int64,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>)
   at System.Threading.Tasks.Parallel.ForEach[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Action`1<System.__Canon>)
   at Microsoft.CST.OAT.Analyzer+<GetCaptures>d__22.MoveNext()
   at Microsoft.ApplicationInspector.RulesEngine.RuleProcessor.AnalyzeFile(Microsoft.ApplicationInspector.RulesEngine.TextContainer, Microsoft.CST.RecursiveExtractor.FileEntry, Microsoft.ApplicationInspector.RulesEngine.LanguageInfo, System.Collections.Generic.IEnumerable`1<System.String>, Int32)
   at Microsoft.ApplicationInspector.RulesEngine.RuleProcessor.AnalyzeFile(System.String, Microsoft.CST.RecursiveExtractor.FileEntry, Microsoft.ApplicationInspector.RulesEngine.LanguageInfo, System.Collections.Generic.IEnumerable`1<System.String>, Int32)
   at Microsoft.ApplicationInspector.RulesEngine.RuleProcessor.AnalyzeFile(Microsoft.CST.RecursiveExtractor.FileEntry, Microsoft.ApplicationInspector.RulesEngine.LanguageInfo, System.Collections.Generic.IEnumerable`1<System.String>, Int32)
   at Microsoft.ApplicationInspector.Commands.AnalyzeCommand+<>c__DisplayClass15_1.<PopulateRecords>g__ProcessLambda|2()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
jhutchings1 commented 7 months ago

~Interestingly, this one doesn't reproduce if you enable verbose logging.~

Sorry, that doesn't seem to be true. It didn't reproduce with this command: appinspector analyze -v verbose -s dependabot-core -l ./log.txt

but it did with this one: appinspector analyze -v verbose -s dependabot-core -l ./log.txt -f sarif

gfs commented 7 months ago

Thanks for your report and the follow up info. That might indicate its somehow specifically in the sarif export path. I'll investigate this week.

gfs commented 4 months ago

Forgot to update the ticket, but this was fixed in the codebase with https://github.com/microsoft/ApplicationInspector/commit/e2e13eefa394f794fade9bdd11e7550ad8d36ad0, though I don't think we have a release out with this fix available yet.