search

voltrondata / spark-substrait-gateway

Implements a gateway that speaks the SparkConnect protocol and drives a backend using Substrait (over ADBC Flight SQL).
Apache License 2.0
15 stars 8 forks source link

Support "Config" request to "get_option" with key "spark.sql.session.localRelationCacheThreshold" #61

Closed pthatte1-bb closed 1 month ago

pthatte1-bb commented 1 month ago

SparkConnect's Scala client currently fails for spark.createDataFrame() requests with a NotImplementedError.

The Scala client for this API submits a request to "get_option" with key "localRelationCacheThreshold", and uses the return value to decide whether to send an "AddArtifacts" request for a local_relation, and this type of request currently fails with a NotImplementedError.

SparkConnect code:

Here is the Spark 3.5.1 documentation for the configuration key and its default value - https://spark.apache.org/docs/3.5.1/configuration.html#runtime-sql-configuration image