search

hortonworks-spark / shc

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.
Apache License 2.0
552 stars 281 forks source link

Unable to save data at HBase using SHC #309

Open kp2006it opened 5 years ago

kp2006it commented 5 years ago

I am facing one intermittent issue while saving the data frame to hbase using spark. I am using below package: --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 \ spark version is 2.1

Some times my spark long running job is getting failed use to below exception:

19/04/02 00:02:35 WARN ShutdownHookManager: ShutdownHook '$anon$2' failed, java.util.concurrent.ExecutionException: java.lang.NoClassDefF oundError: org/apache/spark/sql/execution/datasources/hbase/HBaseConnectionCache$$anonfun$resetCache$1 java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/spark/sql/execution/datasources/hbase/HBaseConnection Cache$$anonfun$resetCache$1 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:206) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:67) Caused by: java.lang.NoClassDefFoundError: org/apache/spark/sql/execution/datasources/hbase/HBaseConnectionCache$$anonfun$resetCache$1 at org.apache.spark.sql.execution.datasources.hbase.HBaseConnectionCache$.resetCache(HBaseConnectionCache.scala:86) at org.apache.spark.sql.execution.datasources.hbase.HBaseConnectionCache$.close(HBaseConnectionCache.scala:97) at org.apache.spark.sql.execution.datasources.hbase.HBaseTableScanRDD$$anonfun$getPartitions$1.apply$mcV$sp(HBaseTableScan.scala: 82)