search

stackabletech / spark-k8s-operator

Operator for Apache Spark-on-Kubernetes for Stackable Data Platform
https://stackable.tech
Other
49 stars 2 forks source link

fix: reinstate setting for spark.executor.cores #300

Closed adwk67 closed 8 months ago

adwk67 commented 9 months ago

The setting for spark.executor.cores should be re-instated as is not enough to rely on spark.kubernetes.executor.cores.

From here:

The cores property controls the number of concurrent tasks an executor can run. --executor-cores 5 means that each executor can run a maximum of five tasks at the same time.

If only spark.kubernetes.executor.cores is set, then the request in the pod definition is correct, but the executor application itself does not know how many concurrent tasks can be run.

lfrancke commented 8 months ago

Is this a breaking/CRD change?

adwk67 commented 8 months ago

no, just an internal code change