apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.66k stars 1.7k forks source link

[Bug] [Hive Connector] Hive Connector Kerberos auto Failed #6935

Closed EricJoy2048 closed 2 weeks ago

EricJoy2048 commented 1 month ago

Search before asking

What happened

image

SeaTunnel Version

2.3.5

SeaTunnel Config

env {
  job.mode = "BATCH"
  parallelism = 4
}
source {
  Jdbc {
    url = "jdbc:mysql://datasource01:3306/test?serverTimezone=GMT%2b8"
    driver = "com.mysql.cj.jdbc.Driver"
    connection_check_timeout_sec = 100
    user = "root"
    password = "root@123"

    table_list = [
      {
        table_path = "qa_source.batch_mysql_to_doris"
      },
      {
        table_path = "qa_source.auto_stream_mysql"
        # Use query filetr rows & columns
        query = "select id, c_02_integer_col from qa_source.auto_stream_mysql"
      }
    ]
    #where_condition= "where id > 100"
    #split.size = 8096
    #split.even-distribution.factor.upper-bound = 100
    #split.even-distribution.factor.lower-bound = 0.05
    #split.sample-sharding.threshold = 1000
    #split.inverse-sampling.rate = 1000
  }
}

sink {
  Hive {
    table_name = "gaojun_test.${table_name}"
    metastore_uri = "thrift://datasource01:9083"
    krb5_path=/etc/krb5.conf
    kerberos_principal="hadoop/datasource01@TEST.COM"
    kerberos_keytab_path="/etc/hadoop/conf/hadoop.keytab"
    hdfs_site_path="/etc/hadoop/conf/hdfs-site.xml"
    hive_site_path="/etc/hive/conf/hive-site.xml"
    hive.hadoop.conf={
        hadoop.security.authentication="kerberos"      
    }
  }
}

Running Command

sh seatunnel.sh -c xxx.conf

Error Exception

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:206)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a sink for identifier 'Hive'.
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSink(FactoryUtil.java:130)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.createSinkAction(MultipleTableJobConfigParser.java:638)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:569)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:216)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:105)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:173)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:153)
        ... 2 more
Caused by: org.apache.seatunnel.connectors.seatunnel.hive.exception.HiveConnectorException: ErrorCode:[HIVE-02], ErrorDescription:[Initialize hive metastore client failed] - Login form kerberos failed
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:96)
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:107)
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveTableUtils.getTableInfo(HiveTableUtils.java:41)
        at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.getTableInformation(HiveSink.java:237)
        at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.<init>(HiveSink.java:85)
        at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSinkFactory.lambda$createSink$0(HiveSinkFactory.java:69)
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSink(FactoryUtil.java:127)
        ... 8 more
Caused by: java.lang.IllegalArgumentException: hadoop.security.authentication must be kerberos
        at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopLoginFactory.loginWithKerberos(HadoopLoginFactory.java:38)
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:60)
        ... 14 more

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 3 weeks ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 2 weeks ago

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.