Open suchitgupta01 opened 3 years ago
@suchitgupta01 I am facing a problem with something you seem to have solved.
My log4j.properties file is present at /app/spark/conf/log4j.properties
and I have added the following code in the sparkConf: new SparkConf().set("spark.executor.extraJavaOptions", "Dlog4j.configuration=/app/spark/conf/log4j.properties")
But it looks like spark is unable to locate it. What could be the problem here? Any idea?
Hi Team,
I was trying to override log4j for drivers and executors by creating a config map. The logging for drivers works fine but for executors it was taking a default configuration.
I created a config map and added
sparkConf: spark.driver.extraJavaOptions: -Dlog4j.configuration=file:/etc/spark/conf/log4j.properties spark.executor.extraJavaOptions: -Dlog4j.configuration=file:/etc/spark/conf/log4j.properties
With the changes driver logs were as per custom log4j properties but executor logs were taking default log4j.
On investigating found that If I remove the
.spec.template.executor.javaOptions
then executors logs take custom log4j.Is this expected behavior?
My yaml file:
Spark: 3.0.0 Scala: 2.12