feat: support get_option for local-rel threshold

voltrondata / spark-substrait-gateway

Implements a gateway that speaks the SparkConnect protocol and drives a backend using Substrait (over ADBC Flight SQL).

Apache License 2.0

15 stars 8 forks source link

feat: support get_option for local-rel threshold #65

Closed pat70 closed 1 month ago

pat70 commented 1 month ago

SparkConnect's Scala client currently fails for spark.createDataFrame() requests with a NotImplementedError.

The Scala client for this API submits a request to "get_option" with key "localRelationCacheThreshold". We only support ConfigRequests for this key if "op_type" is "get".

fixes #61

github-actions[bot] commented 1 month ago

ACTION NEEDED

Substrait follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

EpsilonPrime commented 1 month ago

Thanks for the fix!