opensearch-project / opensearch-spark

Spark Accelerator framework ; It enables secondary indices to remote data stores.
Apache License 2.0
12 stars 18 forks source link

[FEATURE] REPL concurrent query execution #382

Open noCharger opened 2 weeks ago

noCharger commented 2 weeks ago

Is your feature request related to a problem?

https://github.com/opensearch-project/opensearch-spark/blob/main/spark-sql-application/src/main/scala/org/apache/spark/sql/FlintREPL.scala#L814-L825

      val futureQueryExecution = Future {
        executeQuery(
          spark,
          flintStatement.query,
          dataSource,
          flintStatement.queryId,
          sessionId,
          false)
      }(executionContext)

      // time out after 10 minutes
      ThreadUtils.awaitResult(futureQueryExecution, queryExecutionTimeOut)

Currrently REPL query loop will wait on async query execution.

What solution would you like?

Need some thoughts around to support REPL concurrent query execution.

What alternatives have you considered?

Concurrent execution outside of Spark. The problem of this approach is code start.