search

github / codeql

CodeQL: the libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security
https://codeql.github.com
MIT License
7.66k stars 1.53k forks source link

query keep waiting :An error occurred while evaluating _BasicBlocks #15440

Closed HeouDonkey closed 2 weeks ago

HeouDonkey commented 9 months ago

when i try to analysis aosp database with this qury script:

/**
 * @id 1
 * @kind path-problem
 */
import java
import semmle.code.java.dataflow.FlowSources
import semmle.code.java.dataflow.TaintTracking

 module SensitiveLoggerConfig implements DataFlow::ConfigSig {  // 1: module always implements DataFlow::ConfigSig or DataFlow::StateConfigSig
  predicate isSource(DataFlow::Node source) {    
  exists(Method method |
    method.hasName("updateSubscriptionInfoByIccId") 
    and source.asParameter() = method.getParameter(0)
    ) } // 3: no need to specify 'override'
  predicate isSink(DataFlow::Node sink) { 
    exists(Method method |
      method.hasName("addSubscriptionInfoRecord") 
      and sink.asParameter() = method.getParameter(1)
      )
  }

  // predicate isBarrier(DataFlow::Node sanitizer) {  // 4: 'isBarrier' replaces 'isSanitizer'
  // }

  // predicate isBarrierIn(DataFlow::Node node) { isSource(node) } // 4: isBarrierIn instead of isSanitizerIn
}

module SensitiveLoggerFlow = TaintTracking::Global<SensitiveLoggerConfig>; // 2: TaintTracking selected 

import SensitiveLoggerFlow::PathGraph  // 7: the PathGraph specific to the module you are using

from SensitiveLoggerFlow::PathNode source, SensitiveLoggerFlow::PathNode sink  // 8 & 9: using the module directly
where SensitiveLoggerFlow::flowPath(source, sink)  // 9: using the flowPath from the module 
select sink.getNode(), source, sink, "test", source.getNode(),
  "test"

something wrong with this query,the query server log is :

[2024-01-25 23:48:08] [ERROR] Working on: 
                                  com.semmle.inmemory.scheduler.IntensionalLayer$IntensionalWork (state=WORKING_RECRUITING, 0 working threads)
                                  Saw exception: CatastrophicError: An error occurred while evaluating _BasicBlocks::BasicBlock.getABBSuccessor/0#dispred#27e6c59b_BasicBlocks::BasicBlock.getABBSuccessor/__#antijoin_rhs/2@961dff9c
[2024-01-25 23:48:08] [ERROR] Exception collected asynchronously; saving it while cancelling everything
                              com.semmle.util.exception.CatastrophicError: An error occurred while evaluating _BasicBlocks::BasicBlock.getABBSuccessor/0#dispred#27e6c59b_BasicBlocks::BasicBlock.getABBSuccessor/__#antijoin_rhs/2@961dff9c
                              com.semmle.util.exception.CatastrophicError: Could not load page 0 of BasicBlocks::BasicBlock.getABBSuccessor/0#dispred#27e6c59b/2@151100iu (233258gp6el35eq9tdcj9hfru7f2o)
                              The RA to evaluate was:

                                  {2} r1 = JOIN `Dominance::bbIDominates/2#5b2d0096` WITH `BasicBlocks::BasicBlock.getABBSuccessor/0#dispred#27e6c59b` ON FIRST 2 OUTPUT Lhs.1, Lhs.0
                                  {3} r2 = JOIN r1 WITH `BasicBlocks::BasicBlock.getABBSuccessor/0#dispred#27e6c59b_10#join_rhs` ON FIRST 1 OUTPUT Lhs.1, Lhs.0, Rhs.1
                                  {3} r3 = REWRITE r2 WITH TEST InOut.0 != InOut.2
                                  {3} r4 = SCAN r3 OUTPUT In.1, In.2, In.0
                                  {3} r5 = r4 AND NOT `Dominance::bbDominates/2#3865e3dd`(FIRST 2)
                                  {2} r6 = SCAN r5 OUTPUT In.2, In.0
                                  return r6

                              (eventual cause: CatastrophicError "Could not load page 0 of BasicBlocks::BasicBlock.getABBSuccessor/0#dispred#27e6c...")
                                at com.semmle.inmemory.pipeline.PipelineInstance.wrapWithRaDump(PipelineInstance.java:168)
                                at com.semmle.inmemory.pipeline.PipelineInstance.exceptionCaught(PipelineInstance.java:152)
                                at com.semmle.inmemory.scheduler.execution.ThreadableWork.handleAndLog(ThreadableWork.java:549)
                                at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:373)
                                at com.semmle.inmemory.scheduler.IntensionalLayer$IntensionalWork.evaluate(IntensionalLayer.java:71)
                                at com.semmle.inmemory.scheduler.SimpleLayerTask$SimpleLayerWork.doWork(SimpleLayerTask.java:69)
                                at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:359)
                                at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:601)
                                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                                at java.base/java.lang.Thread.run(Unknown Source)
                              Caused by: com.semmle.util.exception.CatastrophicError: Could not load page 0 of BasicBlocks::BasicBlock.getABBSuccessor/0#dispred#27e6c59b/2@151100iu (233258gp6el35eq9tdcj9hfru7f2o)
                                at com.semmle.inmemory.caching.PagedRelation.getPage(PagedRelation.java:95)
                                at com.semmle.inmemory.relations.IntArrayMatcher.switchPageTo(IntArrayMatcher.java:151)
                                at com.semmle.inmemory.relations.IntArrayMatcher.startMatching(IntArrayMatcher.java:624)
                                at com.semmle.inmemory.relations.IntArrayMatcher.testBatch(IntArrayMatcher.java:655)
                                at com.semmle.inmemory.pipeline.SetOpStep$SetOpState.processBatch(SetOpStep.java:75)
                                at com.semmle.inmemory.pipeline.SimpleStep$SimpleState.deliverBatch(SimpleStep.java:139)
                                at com.semmle.inmemory.pipeline.SimpleStep$SimpleState.deliverOwnBatch(SimpleStep.java:124)
                                at com.semmle.inmemory.pipeline.ExpanderStep$ExpanderState.deliverMatchOutput(ExpanderStep.java:71)
                                at com.semmle.inmemory.pipeline.LiteralStep$LiteralState.deliverRelation(LiteralStep.java:68)
                                at com.semmle.inmemory.pipeline.LiteralStep.deliverRelation(LiteralStep.java:139)
                                at com.semmle.inmemory.pipeline.LiteralStep$1.headEndWork(LiteralStep.java:126)
                                at com.semmle.inmemory.pipeline.HeadEndDispatcher.headEndWork(HeadEndDispatcher.java:75)
                                at com.semmle.inmemory.pipeline.PipelineState.doSomeWork(PipelineState.java:78)
                                at com.semmle.inmemory.pipeline.PipelineInstance.doWork(PipelineInstance.java:117)
                                at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:359)
                                ... 7 common frames omitted
[2024-01-25 23:48:08] (1293s) Cancelling taintAnalysisTest.ql with a UnhandledAsyncException and message CatastrophicError caught in asynchronous computation.
[2024-01-25 23:48:08] (1293s) Query done
[2024-01-25 23:48:08] (1293s) Pausing evaluation to sync to disk at sequence stamp o+9594333
[2024-01-25 23:48:08] (1293s) Unpausing evaluation
[2024-01-25 23:48:08] [WARN] Evaluation of /home/houanran/codeql/ql/java/ql/examples/taintAnalysisTest.ql terminated abnormally.
                             java.util.concurrent.CompletionException: com.semmle.util.concurrent.UnhandledAsyncException: 1 asynchronous exceptions caught
                                at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
                                at com.semmle.inmemory.scheduler.CapstoneTask$1.doWork(CapstoneTask.java:97)
                                at com.semmle.util.concurrent.RateLimitedAction.wokenUp(RateLimitedAction.java:76)
                                at com.semmle.util.concurrent.RateLimitedAction.access$300(RateLimitedAction.java:15)
                                at com.semmle.util.concurrent.RateLimitedAction$BackgroundThread.run(RateLimitedAction.java:196)
                             Caused by: com.semmle.util.concurrent.UnhandledAsyncException: 1 asynchronous exceptions caught
                                at com.semmle.util.concurrent.CancellationToken.getException(CancellationToken.java:182)
                                at com.semmle.inmemory.scheduler.QueryEvaluator.whenCancelled(QueryEvaluator.java:381)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken$SubCancelListener.cancelled(CancellationToken.java:200)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken$SubCancelListener.cancelled(CancellationToken.java:200)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken$SubCancelListener.cancelled(CancellationToken.java:200)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:137)
                                at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:631)
                                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                                at java.base/java.lang.Thread.run(Unknown Source)
[2024-01-25 23:48:08] [PROGRESS] execute query-server2> Error running query
[2024-01-25 23:48:08] [ERROR] Error running query
                              java.util.concurrent.CompletionException: com.semmle.util.concurrent.UnhandledAsyncException: 1 asynchronous exceptions caught
                                at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
                                at com.semmle.inmemory.scheduler.CapstoneTask$1.doWork(CapstoneTask.java:97)
                                at com.semmle.util.concurrent.RateLimitedAction.wokenUp(RateLimitedAction.java:76)
                                at com.semmle.util.concurrent.RateLimitedAction.access$300(RateLimitedAction.java:15)
                                at com.semmle.util.concurrent.RateLimitedAction$BackgroundThread.run(RateLimitedAction.java:196)
                              Caused by: com.semmle.util.concurrent.UnhandledAsyncException: 1 asynchronous exceptions caught
                                at com.semmle.util.concurrent.CancellationToken.getException(CancellationToken.java:182)
                                at com.semmle.inmemory.scheduler.QueryEvaluator.whenCancelled(QueryEvaluator.java:381)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken$SubCancelListener.cancelled(CancellationToken.java:200)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken$SubCancelListener.cancelled(CancellationToken.java:200)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken$SubCancelListener.cancelled(CancellationToken.java:200)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:137)
                                at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:631)
                                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                                at java.base/java.lang.Thread.run(Unknown Source)
[2024-01-25 23:48:08] [PROGRESS] execute query-server2> Error running query
[2024-01-25 23:48:08] [ERROR] Error running query
                              java.util.concurrent.CompletionException: com.semmle.util.concurrent.UnhandledAsyncException: 1 asynchronous exceptions caught
                                at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
                                at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
                                at com.semmle.inmemory.scheduler.CapstoneTask$1.doWork(CapstoneTask.java:97)
                                at com.semmle.util.concurrent.RateLimitedAction.wokenUp(RateLimitedAction.java:76)
                                at com.semmle.util.concurrent.RateLimitedAction.access$300(RateLimitedAction.java:15)
                                at com.semmle.util.concurrent.RateLimitedAction$BackgroundThread.run(RateLimitedAction.java:196)
                              Caused by: com.semmle.util.concurrent.UnhandledAsyncException: 1 asynchronous exceptions caught
                                at com.semmle.util.concurrent.CancellationToken.getException(CancellationToken.java:182)
                                at com.semmle.inmemory.scheduler.QueryEvaluator.whenCancelled(QueryEvaluator.java:381)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken$SubCancelListener.cancelled(CancellationToken.java:200)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken$SubCancelListener.cancelled(CancellationToken.java:200)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken$SubCancelListener.cancelled(CancellationToken.java:200)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:86)
                                at com.semmle.util.concurrent.CancellationToken.cancel(CancellationToken.java:137)
                                at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:631)
                                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                                at java.base/java.lang.Thread.run(Unknown Source)
hmakholm commented 9 months ago

Thanks for the report. This is a rarely occurring situation where some internal structures in the QL evaluator somehow get into an inconsistent state. It usually clears up when you try to rerun the query.

Unfortunately, by the time the inconsistency is detected, there's not really any information left about what actually went wrong anymore. Combined with the rarity of the symptom, and the fact that it seems to be nondeterministic and timing-dependent, we have not yet been able to find the underlying bug. If you come across a database and query that reproduces it consistently, we'd be very interested in taking a look at that.

(For what it's worth, the characteristic symptom here is com.semmle.util.exception.CatastrophicError: Could not load page <number> of <blah> halfway down the stack traces you quote).

HeouDonkey commented 8 months ago

i found that when i finilize a database and query with taint analysis the first time,this query will work will, but after that, i query the same ql ,it wents wrong with the same error , is the query process changed something in the database?

MasoudAbdaal commented 5 months ago

I got the same error and while the query got excecption, my RAM remained full and did not clean up! After when I tried to run another query, I got the following:

CodeQL Query:

import codeql.ruby.AST
import codeql.ruby.DataFlow
import codeql.ruby.security.HttpToFileAccessQuery
import HttpToFileAccessFlow::PathGraph

from HttpToFileAccessFlow::PathNode source, HttpToFileAccessFlow::PathNode sink
where HttpToFileAccessFlow::flowPath(source, sink)
select sink.getNode(), source, sink, "Write to file system depends on $@.", source.getNode(),
  "untrusted data"

CodeQL Query Server Output:

[2024-05-08 04:38:32] [ERROR] Error running query java.lang.OutOfMemoryError: Java heap space at com.semmle.bdd.SemmleBDDFactory.(SemmleBDDFactory.java:290) at com.semmle.dil.proptypes.QLPropTypeHierarchy.createHierarchy(QLPropTypeHierarchy.java:84)

hmakholm commented 2 weeks ago

We have fixed a bug in the cache management subsystem of the QL evaluator that was almost surely the cause of the com.semmle.util.exception.CatastrophicError: Could not load page 0 crash.

The fix ought to be included in the next release of the CodeQL CLI.