Spark 2 Support

[tonyfraser]

I had to compile spark-sftp by hand. It doesn't SBT in, I get the same missing library errors in the other issue.

After I added it to my classpath,

val myDf = spark.read
 .format("com.springml.spark.sftp")
 .option("host", "{}")
 .option("username", "{}")
 .option("password", "{}")
 .option("fileType", "csv")
.load("/path/to/remote copy.csv.gz")
.repartition(20)

It threw this error.

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.DataFrameReader.load(Ljava/lang/String;)Lorg/apache/spark/sql/DataFrame;
    at com.springml.spark.sftp.DatasetRelation.read(DatasetRelation.scala:44)
    at com.springml.spark.sftp.DatasetRelation.<init>(DatasetRelation.scala:29)
    at com.springml.spark.sftp.DefaultSource.createRelation(DefaultSource.scala:82)
    at com.springml.spark.sftp.DefaultSource.createRelation(DefaultSource.scala:43)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:315)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:132)

I changed this in build.sbt.

scalaVersion := "2.11.8"
sparkVersion := "2.0.0"

Added this in plugins.sbt

resolvers += "Spark Package Main Repo" at "https://dl.bintray.com/spark-packages/maven"

Changed this DataSource.scala

//df.coalesce(1).write.avro(tempLocation)

And it compiled and ran fine. Thanks for an amazing plugin you guys!

[samuel-pt]

Thanks for a lot for the information. Would it be possible for you to create a pull request with your change?

[somnanda]

@tonyfraser Could you pull your changes for Spark 2.0 support?

[somnanda]

@samuel-pt Is there an update for Spark 2.0.2 for this project?

[samuel-pt]

@somnanda If @tonyfraser provides me a pull request I can merge the changes into it.

[springml]

@tonyfraser @somnanda This issue is resolved by this commit