It appears that when trying to write a dataframe to an sftp server, only the crc file gets copied. Here's a line from the log:
18/09/19 16:34:47 INFO DefaultSource: Copying C:\Users\jschott\AppData\Local\Temp\spark_sftp_connection_temp931.part-00000-86842b85-b582-441c-9d7f-7b4021efd734-c000.csv.crc to /tmp/sample.csv
Here is the code:
df.write.format("com.springml.spark.sftp").
option("host", "cdtsqbm100d.rxcorp.com").
option("username", "****").
option("password", "****").
option("fileType", "csv").
option("delimiter", ";").
save("/tmp/sample.csv")
It appears that when trying to write a dataframe to an sftp server, only the crc file gets copied. Here's a line from the log: 18/09/19 16:34:47 INFO DefaultSource: Copying C:\Users\jschott\AppData\Local\Temp\spark_sftp_connection_temp931.part-00000-86842b85-b582-441c-9d7f-7b4021efd734-c000.csv.crc to /tmp/sample.csv
Here is the code: df.write.format("com.springml.spark.sftp"). option("host", "cdtsqbm100d.rxcorp.com"). option("username", "****"). option("password", "****"). option("fileType", "csv"). option("delimiter", ";"). save("/tmp/sample.csv")
Here is my build.sbt:
name := "testFTP" version := "0.1" scalaVersion := "2.11.6" val sparkVersion = "2.2.0" libraryDependencies += "com.springml" % "spark-sftp_2.11" % "1.1.1" libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion, "org.apache.spark" %% "spark-sql" % sparkVersion, //"org.apache.spark" %% "spark-mllib" % sparkVersion, //"org.apache.spark" %% "spark-streaming" % sparkVersion, "org.apache.spark" %% "spark-hive" % sparkVersion )
I saw a closed issue and tried building source code myself an including that JAR, but got the same results.