search

hortonworks-spark / shc

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.
Apache License 2.0
552 stars 281 forks source link

Authentication Issues and jar issue #146

Open sa255304 opened 7 years ago

sa255304 commented 7 years ago

1)For the below security credentials manager as provided in the SHC github page like below. spark.hbase.connector.security.credentials ambari-qa-c1@EXAMPLE.COM spark.hbase.connector.security.keytab /etc/security/keytabs/smokeuser.headless.keytab Which Security keytab i need to Provide. I have 3 types of security keytabs and principals. One is hbase.headless.keytab 2nd one is spark.headless.keytab 3rd oneis normal keytab my project acount its like PROJECTNAME.keytab I am facing this Authentication issue in old way of connecting to Hbase also.

2) And in Hortonworks blog for the Hbase Connector there is a different package name given to use.

pyspark –packages zhzhan:shc:0.0.11-1.6.1-s_2.10

pyspark --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 --repositories In the above two what is correct Jar to be used.

weiqingy commented 7 years ago

1) It depends on what user account you were using to run your applications. So here, I guess you may want to use PROJECTNAME.keytab.

2) It depends on which Spark version you want to run with. zhzhan:shc:0.0.11-1.6.1-s_2.10 is a very old version, which can run against Spark 1.6.1. com.hortonworks:shc-core:1.1.1-2.1-s_2.11 is latest version, which can run against Spark 2.1. com.hortonworks:shc-core:1.1.1-1.6-s_2.10 is for Spark 1.6.x. We encourage users to try the packages uploaded in Hortonworks repo.