AVSystem / scex

Extensible, fast and secure Scala expression evaluation engine
MIT License
21 stars 4 forks source link

Completer.getErrors gives quite a lot of false positives #17

Open najder-k opened 2 years ago

najder-k commented 2 years ago

Example code:

object ScexFlakyValidation {
  private class DefaultJavaScexCompiler(val settings: ScexSettings)
    extends ScexCompiler
      with ScexPresentationCompiler
      with ClassfileReusingScexCompiler
      with TemplateOptimizingScexCompiler
      with CachingScexCompiler // removing this fixes it, but requires a lower count of tests to not crash (1000 is too much)
      with CachingScexPresentationCompiler
      with WeakReferenceWrappingScexCompiler
      with JavaScexCompiler

  def main(args: Array[String]): Unit = {
    val symbolValidator = SymbolValidator(PredefinedAccessSpecs.basicOperations)
    val symbolAttributes = SymbolAttributes(Nil)
    val syntaxValidator = SyntaxValidator.SimpleExpressions
    val utils = NamedSource("test", "")
    val profile = new ExpressionProfile("test", syntaxValidator, symbolValidator, symbolAttributes, "", utils)
    val settings = new ScexSettings
//    settings.noGetterAdapters.value = true //adding this option fixes it too
    val scexCompiler = new DefaultJavaScexCompiler(settings)
    val completer = scexCompiler.getCompleter[SimpleContext[Any], String](profile, template = false)

    def validateExpr(expr: String): Boolean = completer.getErrors(expr).isEmpty

    val invalidExpression = "symbolMissing" //basically any invalid expression works

    val totalCount = 1000
    val fpCount = (0 until totalCount)
      .map(i => invalidExpression + " " * i) //spaces added to avoid hitting the cache
      .count(validateExpr) // <- this is what gives false positives sometimes (completer.getErrors)

    if (fpCount != 0) {
      println(s"There were $fpCount/$totalCount false positives in validation")
    } else {
      println("all good")
    }
  }
}

This code tries to check for errors on the same invalid expression 1000 times (with added spaces to avoid hitting the cache). The number and order of FPs is nondeterministic, and seems to get higher the longer the code runs, e.g. for 1k times it's around 50-70%:

There were 695/1000 false positives in validation

But for 100 times it's between 10-30%:

There were 12/100 false positives in validation

It seems to not happen with either setting the noGetterAdapters = true in ScexSettings, or when using a ScexCompiler without CachingScexCompiler mixed in, even though those seem to be completely unrelated.

Some other findings:

anetaporebska commented 2 months ago

It seems that there is a problem with com.avsystem.scex.compiler.ScexCompiler.Reporter#errorsBuilder and the fact that errors are cleared both by scex code and the Scala Presentation Compiler code (scala.tools.nsc.interactive.Global#backgroundCompile) by calling reporter.reset(). The latter runs arbitrarily and sometimes interferes with accumulated errors and clears them.

Since I didn't find any direct connection between cache/getterAdapters and errorsBuilder I assume they affect validation time and thus contribute to different results.

Workaround is presented here: https://github.com/AVSystem/scex/pull/36