search

JoeShi / kafka-realtime-data-processing

MIT License
0 stars 2 forks source link

Hue submit Spark task file in S3 will result ClassNotFound error #17

Open JoeShi opened 4 years ago

JoeShi commented 4 years ago

Use the following configuration to choose file from S3. Screen Shot 2019-11-18 at 10 55 23 AM

Error: Screen Shot 2019-11-18 at 10 56 06 AM

If choose from HDFS, there is no error.

JoeShi commented 4 years ago

tried the following command, and failed

sudo ln -s /usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.36.0.jar ./emrfs-hadoop-assembly.jar
JoeShi commented 4 years ago

Copy lib emrfs to oozie lib. and change s3a to s3 will resolve this issue.

COPY EMRFS lib jar to oozie lib directory.

sudo cp /usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-*.jar /usr/lib/oozie/lib/
sudo stop oozie
sudo start oozie

Change s3a to s3 Screen Shot 2019-11-18 at 12 04 06 PM