Closed JustAnotherContributor closed 1 year ago
JustAnotherContributor
commented
1 year ago As I'm using EMR 6.4.0, I also tried the following dependencies but it didn't help:
kdgregory
commented
1 year ago @JustAnotherContributor - sorry, I can't provide much help with this, as it's an issue with application packaging, and not the library itself.
I will note that the Spark documentation says to package any dependencies in an "uber" JAR (https://spark.apache.org/docs/latest/submitting-applications.html), so I would start there.
JustAnotherContributor
commented
1 year ago I tried many different ways of getting this to work (I was already using an "uber" JAR with the appenders JAR and its dependencies specified in my app's POM) and, while not ideal, the only working solution I could find was to programmatically add the appender within the EMR step's Java code:
import com.kdgregory.log4j.aws.CloudWatchAppender;
import org.apache.log4j.PatternLayout;
import org.apache.log4j.Logger;
...
CloudWatchAppender cloudwatch = new CloudWatchAppender();
cloudwatch.setLogGroup("{env:APP_NAME}-{sysprop:deployment:dev}");
cloudwatch.setLogStream("{hostname}-{startupTimestamp}");
cloudwatch.setDedicatedWriter(true);
cloudwatch.setLayout(new PatternLayout("%d [%t] %-5p - %c - %m%n"));
cloudwatch.activateOptions();
Logger.getRootLogger().addAppender(cloudwatch);For anyone that comes across this issue, I was able to get things working by adding a bootstrap script with the following to the EMR cluster:
echo "installing cloudwatch log4jv1 appender"
sudo mkdir -p /usr/lib/spark/jars/
for artifact in "aws-facade-v1" "log4j1-aws-appenders" "logwriters"; do sudo wget https://repo1.maven.org/maven2/com/kdgregory/logging/$artifact/3.0.1/$artifact-3.0.1.jar -O /usr/lib/spark/jars/$artifact-3.0.1.jar; doneThen the method I was using to provide the log4j.xml (or log4j.properties) file works:
--files,s3a://<bucket>/<folder>/log4j.xml,--conf,spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j.xml,--conf,spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.xmlNote: the com.amazonaws:aws-java-sdk-log artifact is not required as it already exists on the EMR cluster nodes.
When I run my Java app locally, the CloudWatch Appender works fine but when I upload my JAR to an EMR cluster the Spark step is unable to find/load the class. I have the following packages specified as dependencies within my app's POM and have also tried specifying them using the --packages option to the spark submit command, in the case of the latter I can see the spark step downloading the packages and their dependencies and uploading them to the driver and executors so they definitely should be available in the classpath. Any assistance would be appreciated as I suspect I'm overlooking something, most likely simple.