search

ExpediaGroup / waggle-dance

Hive federation service. Enables disparate tables to be concurrently accessed across multiple Hive deployments.
Apache License 2.0
274 stars 76 forks source link

Add personalized configuration parameters for each metastore. #315

Closed yangyuxia closed 5 months ago

yangyuxia commented 7 months ago

:pencil: Description

Different hive Metastores may have different parameter values for the same parameter. If multiple federated Metastores are configured in waggle dance, you can configure their own parameter values for each metastore. For example, when hivemetastore enabled the kerberos authentication, the value of the "hive.metastore.kerberos.principal", some metastore is configured to "hive/_HOST@Realm", but some metastore is configured as "hive/{clusterName}@Realm", so you need to configure personalized configuration parameters for each metastore.

:link: Related Issues

yangyuxia commented 7 months ago

cat waggle-dance-server.yml

verbose: true port: 48869 database-resolution: PREFIXED yaml-storage:  overwrite-config-on-shutdown: false logging:  config: file:${WAGGLE_DANCE_HOME}/conf/log4j2.xml configuration-properties:  hadoop.security.authentication: KERBEROS  hadoop.kerberos.keytab.login.autorenewal.enabled: true  hadoop.proxyuser.hive.users: ''  hadoop.proxyuser.hive.groups: ''  hadoop.proxyuser.hive.hosts: '*'  hive.metastore.sasl.enabled: true  hive.metastore.kerberos.principal: hive/_HOST@HADOOP.COM  hive.metastore.kerberos.keytab.file: ${WAGGLE_DANCE_HOME}/conf/wd.keytab  hive.cluster.delegation.token.store.class: org.apache.hadoop.hive.thrift.ZooKeeperTokenStore  hive.cluster.delegation.token.store.zookeeper.connectString: host-zk1:2181,host-zk2:2181,host-zk3:2181  hive.cluster.delegation.token.store.zookeeper.znode: /hive/cluster/delegationwg  hive.cluster.delegation.token.store.zookeeper.acl: "world:anyone:rwcda"  hive.server2.authentication: KERBEROS  hive.server2.authentication.kerberos.keytab: ${WAGGLE_DANCE_HOME}/conf/wd.keytab  hive.server2.authentication.kerberos.principal: hive/_HOST@HADOOP.COM  hive.server2.authentication.client.kerberos.principal: hive/_HOST@HADOOP.COM  wd-metastore:   hms1:     hive.metastore.kerberos.principal: hive/_HOST@HADOOP.COM   hms2:     hive.metastore.kerberos.principal: hive/test2@HADOOP.COM

yangyuxia commented 7 months ago

primary-meta-store:   database-prefix: ''   name: hms1   remote-meta-store-uris: thrift://host-hms1:9083   access-control-type: READ_AND_WRITE_AND_CREATE   impersonation-enabled: false   configuration-properties:     hive.metastore.kerberos.principal: hive/hms@HADOOP.COM     hive.server2.enable.doAs: true

jmnunezizu commented 6 months ago

Thanks for the contribution. Please update the readme explaining how this works and how it can be utilised. Thanks.