temp location hdfs

[mittalakhilesh]

Hi , How can we provide temp location to be hdfs?

[springml]

@mittalakhilesh We've added new option hdfsTempLocation to specify temp location in hdfs. You can use it in addition to local temp location. Like,

df.write.
      format("com.springml.spark.sftp").
      option("host", "sftp_host").
      option("username", "username
      option("password", "password").
      option("fileType", "csv").
      option("delimiter", ";").
      option("hdfsTempLocation", "/tmp/sftp").
      option("tempLocation", "/tmp").            
      save("/target_location/target.csv")

Please note that this is not yet pushed to maven repository. If needed, build from source code and include it in spark using --jars option

[mittalakhilesh]

Thanks...