search

springml / spark-sftp

Spark connector for SFTP
Apache License 2.0
100 stars 98 forks source link

EMR deployment error #91

Open snelluri1277 opened 2 years ago

snelluri1277 commented 2 years ago

the code is working fine at my local machine. but, I am getting an error while running in EMR cluster. could you please help to resolve the issue. Exception: org.apache.hadoop.security.AccessControlException: Permission denied: user=hive, access=WRITE, inode="/":hdfs:hadoop:drwxr-xr-x

below is my code snippet

val df = ss.read.format("csv"). format("com.springml.spark.sftp"). option("delimiter", "|"). option("host", FTP_Host). option("username", FTP_Username). option("password", FTP_Password). option("fileType", "csv"). option("inferSchema", "true"). option("encoding","UTF-8").
load(FTP_File_Path_final)

pitt-william-t commented 1 year ago

that's a permission issue, nothing to do with this codebase