apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.44k stars 959 forks source link

[Feature] Support table options via sql conf for Spark Engine #4371

Closed xiangyuf closed 2 weeks ago

xiangyuf commented 1 month ago

Search before asking

Motivation

In some cases, users may want to use spark time travel by setting properties like set spark.paimon.scan.tag-name=tag_3. However, this property will take effect globally if the spark job read multiple tables at the same time.

It would be better if we can support table level options via sql conf for Spark Engine.

image

Solution

Support table level options.

Anything else?

No response

Are you willing to submit a PR?