search

hashicorp / nomad-spark

DEPRECATED: Apache Spark with native support for Nomad as a scheduler
44 stars 16 forks source link

Passing Executor extraJavaOptions from Spark Submit #21

Closed sgtang closed 5 years ago

sgtang commented 5 years ago

We've been attempting to collect executor metrics by running a .jar using the following configuration in our Spark submit:

--conf "spark.executor.extraJavaOptions=-javaagent:jmx_prometheus_javaagent-0.2.0.jar=7072:spark.yml"

It seems as if this configuration is being completely ignored by Nomad Spark and not being passed to the executor. We have to manually edit the job plan's SPARK_EXECUTOR_OPTS environment variable to include the javaagent statement to get the metrics to work.

Is there actually a way for us to pass this .jar to the executors from the spark submit without resorting to editing the job .json and doing an update? Moreover, if the Spark job gets killed, the environment variable is overwritten again, so doing it this way isn't very sustainable for us.

cgbaker commented 5 years ago

Which version of spark-nomad are you using?

sgtang commented 5 years ago

We're on spark-2.4.0-bin-nomad-0.8.6-20181220.

cgbaker commented 5 years ago

Thank you. I'll look into this; I added support for spark.driver.extraJavaOptions in this last release, it may be that extra options for the executors are still missing.

georgehdd commented 5 years ago

Any updates on this?

cgbaker commented 5 years ago

hi @georgehdd , thanks for asking. no update as of yet, but i'm in the middle of a new release of nomad-spark with some other fixes/features, and my plan is to roll this in as well.

georgehdd commented 5 years ago

Hi @cgbaker, Any other way to pass extraJavaOptions to executors? I'm trying to change GC settings on executors and adding the spark.executor.extraJavaOptions in the spark-defaults.conf doesn't seem to be affecting either.

cgbaker commented 5 years ago

It may be possible by adding a JAVA_OPTS variable to the executor environment. This can be done by adding the following to the spark-submit command. For example:

  --conf spark.executorEnv.JAVA_OPTS="-XX:-UseParallelGC" \

or, alternatively, adding the environment variable directly to the executor task in the job template (if you are using/willing to use a job template).

georgehdd commented 5 years ago

@cgbaker Tried both (conf parameter and job template) _JAVAOPTS doesn't seem to be having any effect on the executors. Do you have an ETA on the new release which will include the spark.executor.extraJavaOptions fix?

cgbaker commented 5 years ago

I'll try to release this week. Would 2.4.1 be sufficient or do you need 2.4.0?

On Sun, Jul 7, 2019, 02:29 georgehdd notifications@github.com wrote:

@cgbaker https://github.com/cgbaker Tried both (conf parameter and job template) JAVA_OPTS doesn't seem to be having any effect on the executors. Do you have an ETA on the new release which will include the spark.executor.extraJavaOptions fix?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp/nomad-spark/issues/21?email_source=notifications&email_token=AAMY6TZR7LXJGIHJX2WB5BTP6GLNPA5CNFSM4GTFLWV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZLGGOQ#issuecomment-508977978, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMY6T2GLGZEQWSIFF6GOG3P6GLNPANCNFSM4GTFLWVQ .

georgehdd commented 5 years ago

2.4.1 would be sufficient. Thanks!

georgehdd commented 5 years ago

Hi @cgbaker, any updates regarding the release?

cgbaker commented 5 years ago

I added it yesterday, doing testing today in preparation for a release. Btw, I targeted 2.4.3 instead of 2.4.1, because it's the latest.

georgehdd commented 5 years ago

2.4.3 is even better. Thanks. How long does testing usually take?

cgbaker commented 5 years ago

Okay, I modified the Nomad scheduler to honor this config.

E.g., for the following config: --conf spark.executor.extraJavaOptions="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" you can see the executors running like so: java ... -XX:+PrintGCDetails -XX:+PrintGCTimeStamps ...

The release is out. I've changed the versioning semantics, feel free to give feedback. The new release is 2.4.3.0: https://github.com/hashicorp/nomad-spark/releases/tag/v2.4.3.0