Closed alarocca-apixio closed 1 year ago
Hey @alarocca-apixio, could you please check this discussion, I'm trying to regroup all the threads on that topic in one place. Please let me know if the plan works for you, before we can decide to move forward with this specific issue.
@alarocca-apixio @Fleid Any progress on this? I would love to see this added. Happy to help on this one, just commented on the PR.
I asked @JCZuurmond to weigh in on the topic, but we haven't had time to catch-up. Any update on your side Cor?
See PR for update 👆
Is this your first time submitting a feature request?
Describe the feature
Configuration for server_side_parameters already exists in the Spark configuration. However, these are currently unused in Spark "session" mode. We can pass these into SessionConnectionWrapper and then, when building the SparkSession, we can pass them as Spark "config":
Describe alternatives you've considered
We've tried Spark via Thrift, but this doesn't support all of our necessary use cases, such as configuring KMS encryption for the jobs.
Who will this benefit?
This will enable users to flexibly control the SparkSession that is created, giving users the ability to inject any parameters desired into the SparkSession.
Are you interested in contributing this feature?
Yes
Anything else?
No response