aliyun / aliyun-emapreduce-datasources

Extended datasource support for Spark/Hadoop on Aliyun E-MapReduce.
http://www.aliyun.com/product/emapreduce
Artistic License 2.0
168 stars 88 forks source link

使用master-2.x时,sparksql报错, Failed to find data source: org.apache.spark.aliyun.maxcompute.datasource. #187

Closed okuc closed 6 years ago

okuc commented 7 years ago

错误如下,该如何解决呢?谢谢 Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.aliyun.maxcompute.datasource. Please find packages at http://spark.apache.org/third-party-projects.html at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:549) at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:86) at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:86) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:301) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146) at ODPSBySPARK2XSQL$.main(ODPSBySPARK2XSQL.scala:19) at ODPSBySPARK2XSQL.main(ODPSBySPARK2XSQL.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.aliyun.maxcompute.datasource.DefaultSource at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$21$$anonfun$apply$12.apply(DataSource.scala:533) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$21$$anonfun$apply$12.apply(DataSource.scala:533) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$21.apply(DataSource.scala:533) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$21.apply(DataSource.scala:533) at scala.util.Try.orElse(Try.scala:84) at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:533) ... 7 more

uncleGen commented 7 years ago

看起来是缺少依赖,你可以参考这个文档: https://help.aliyun.com/document_detail/56932.html。 也可以参考这个代码:https://github.com/aliyun/aliyun-emapreduce-sdk/blob/master-2.x/examples/src/main/scala/com/aliyun/emr/examples/maxcompute/TestODPSDataSource.scala