search

awslabs / amazon-s3-tagging-spark-util

Apache License 2.0
10 stars 2 forks source link

Can we use the external jar on EMR Cluster and call write method using pyspark ? #2

Closed simranjeet97 closed 2 years ago

simranjeet97 commented 3 years ago

Can we use the external jar on EMR Cluster and call write method using pyspark ?

2 ways ----

  1. Add path in spark-submit of jar files ./bin/spark-submit --jars jarfile.jar spark_script.py or set the enviroment variable SPARK_CLASSPATH

  2. add path in the classpath SPARK_CLASSPATH='/path/jarfile.jar:/path/jarfile.jar' spark_script.py

or else after spark session spark.sparkContext.addPyFile("/path/to/jar/jarfile.jar")

Originally posted by @simranjeet97 in https://github.com/awslabs/amazon-s3-tagging-spark-util/issues/1#issuecomment-904100455