java.lang.ClassNotFoundException: com.springml.spark.sftp.DefaultSource

[kevintrannz]

Hi, I follow the doc but it does not work:

 <dependency>
            <groupId>com.springml</groupId>
            <artifactId>spark-sftp_2.11</artifactId>
            <version>1.1.1</version>
        </dependency>

spark.read().
                format("com.springml.spark.sftp").
                option("host", "xxx").
                option("username", "xxx").
                option("password", "xxx").
                option("fileType", "csv")
                .option("inferSchema", "true")
                .option("tempLocation", "/tmp")
                .load("/data/Report.csv");

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: com.springml.spark.sftp. Please find packages at http://spark.apache.org/third-party-projects.html
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:190)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:174)

Caused by: java.lang.ClassNotFoundException: com.springml.spark.sftp.DefaultSource
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23$$anonfun$apply$15.apply(DataSource.scala:618)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23$$anonfun$apply$15.apply(DataSource.scala:618)
    at scala.util.Try$.apply(Try.scala:192)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23.apply(DataSource.scala:618)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23.apply(DataSource.scala:618)
    at scala.util.Try.orElse(Try.scala:84)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:618)
    ... 3 more

Please give me your idea? Thank you.

[samuel-pt]

Did you add this package into Spark container?

[kevintrannz]

@samuel-pt - How? Note that it runs from source code (not spark-submit)

[samuel-pt]

How are you starting spark? Are you connecting with spark running on cluster? Or just running along with your application?

[shivchikkappa]

Is this issue resolved? am getting the similar class not found error. Jar "spark-sftp_2.11-1.1.3.jar" is copied to /usr/hdp/current/spark2-client/jars and spark cluster restarted.

User class threw exception: java.lang.NoClassDefFoundError: com/springml/sftp/client/SFTPClient at com.springml.spark.sftp.DefaultSource.getSFTPClient(DefaultSource.scala:186) at com.springml.spark.sftp.DefaultSource.createRelation(DefaultSource.scala:122) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)

[lidadaer]

You also need two other jar packages: sftp.client-1.0.3.jar and jsch-0.1.53.jar