Open noCharger opened 4 days ago
Is your feature request related to a problem?
Currently, the query loop sleeps for a fixed 100ms for each batch execution. https://github.com/opensearch-project/opensearch-spark/blob/main/spark-sql-application/src/main/scala/org/apache/spark/sql/FlintREPL.scala#L395
What solution would you like?
Add a new spark configuration to support the customizable query loop frequency.
Is your feature request related to a problem?
Currently, the query loop sleeps for a fixed 100ms for each batch execution. https://github.com/opensearch-project/opensearch-spark/blob/main/spark-sql-application/src/main/scala/org/apache/spark/sql/FlintREPL.scala#L395
What solution would you like?
Add a new spark configuration to support the customizable query loop frequency.