Spark options set by the user are not handled properly

[razvan]

Affected Stackable version

all

Affected Apache Spark-on-Kubernetes version

all

Current and expected behavior

Spark properties that are set by both the user via spec.sparkConf and the operator are not merged properly.

Example: users that need to set Java proxy settings via spark.driver.extraJavaOptionscannot do so without overriding the same property set by the operator.

Possible solution

No response

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

[lfrancke]

Reported on Discord

[sbernauer]

To me personally this sounds like expected behavior. The fields docs state

A map of key/value strings that will be passed directly to spark-submit.

which it seems to do.

It think it's a very hard problem to correctly merge all the Spark properties in the world but let's see :)

I personally would consider this a feature request to make it easier to merge stuff from the user onto the operator generated ones - which sounds like a nice quality of live feature - if possible.

[sbernauer]

Reported on Discord

I could not find it. Can you please paste the report here, so we can understand what exactly is going and maybe also conclude this is a bug?

[lfrancke]

I can't find it either right now, sorry.