search

facebookresearch / DCPerf

DCPerf benchmark suite for hyperscale cloud applications
MIT License
116 stars 5 forks source link

Fix an error passing spark.executor.extraJavaOptions #35

Closed hanblee closed 1 year ago

hanblee commented 1 year ago

to Spark executors. Currently, the format of spark.executor.extraJavaOptions being passed is spark.executor.extraJavaOptions="option1 option2", but the expected format is "spark.executor.extraJavaOptions=option1 option2". As a result, spark.executor.extraJavaOptions is not being consumed by Spark executor processes with the following warning in executor's stderr output: WARN SystemPropertyUtil: Unable to parse the boolean system property

hanblee commented 1 year ago

Full warning in the log:

WARN SystemPropertyUtil: Unable to parse the boolean system property 'java.net.preferIPv4Stack':false -djava.net.preferipv6addresses=true -dscala.usejavacp=true -xx:+preserveframepointer -xx:+unlockdiagnosticvmoptions -xx:freqinlinesize=128 -xx:+unlockexperimentalvmoptions -xx:parallelgcthreads=4 -xx:+useparallelgc -xx:+useparalleloldgc -xx:+perfdisablesharedmem - using the default value: false

facebook-github-bot commented 1 year ago

@excelle08 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 1 year ago

@excelle08 merged this pull request in facebookresearch/DCPerf@8ac2d0c3041e6e6d6c351187b9710bee64abda9a.