search

python-zk / kazoo

Kazoo is a high-level Python library that makes it easier to use Apache Zookeeper.
https://kazoo.readthedocs.io
Apache License 2.0
1.3k stars 387 forks source link

Bug in zookeerper kerberos auth #636

Closed tianchao-haohan closed 8 months ago

tianchao-haohan commented 3 years ago

kerberos Auth should be passed

Auth Failed

Snippet to Reproduce the Problem

client = KazooClient("10.8.8.6:24000,10.8.8.7:24000", sasl_options={"mechanism": "GSSAPI", "service":"zookeeper"})
client.start()

Logs with logging in DEBUG mode

AUTH_FAILED closing: Unknown error: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Clock skew too great', -1765328347))

Specifications

Steps:

  1. login kerberos with kinit -kt keytab_file principle
  2. check kerberos auth status with klist, its ok
  3. run the scripts above

The root cause I found is that: The service send to kerberos is "zookeeper@10.8.8.6" (pure_sasl-0.6.2-py3.6.egg/puresasl/mechanisms.py):

class GSSAPIMechanism(Mechanism):
    name = 'GSSAPI'
    score = 100
    qops = QOP.all

    allows_anonymous = False
    uses_plaintext = False
    active_safe = True

    def __init__(self, sasl, principal=None, **props):
        Mechanism.__init__(self, sasl)
        self.user = None
        self._have_negotiated_details = False
        self.host = self.sasl.host
        self.service = self.sasl.service
        self.principal = principal
        self._fetch_properties('host', 'service')

        krb_service = '@'.join((self.service, self.host))         ##self.host here should be hostname rather than host ip

Actually, the correct krb_service should be "zookeeper/hostName@Realm" or "zookeeper@hostName.Realm". for example, zookeeper@hadoop.hadoop.com

KazooClient should take serverFQDN as the input and propagate to puresasl. serverFQDN - the fully qualified domain name of the server (e.g. “serverhost.example.com”).

tianchao-haohan commented 3 years ago

Workaround:

  1. Input service as "zookeeper@serverFQDN"

  2. Change the code:

    
class GSSAPIMechanism(Mechanism):
name = 'GSSAPI'
score = 100
qops = QOP.all

allows_anonymous = False
uses_plaintext = False
active_safe = True

def __init__(self, sasl, principal=None, **props):
    Mechanism.__init__(self, sasl)
    self.user = None
    self._have_negotiated_details = False
    self.host = self.sasl.host
    self.service = self.sasl.service
    self.principal = principal
    self._fetch_properties('host', 'service')

    krb_service = self.service
    if krb_service.find ('@') <= 0 :
        krb_service = '@'.join((self.service, self.host))
ceache commented 3 years ago

Clock skew is too great looks like a time synchronization error, not a bad principal issue. Could you double check that this is the actual error you get from the SASL/Kerberos libraries?

Separately, to help us reproduce this issue, could you please detail the exact configuration of your setup?

Cheers,

On Thu, Jan 21, 2021, 22:37 tianchao-haohan notifications@github.com wrote:

Workaround:

  1. Input service as "zookeeper@serverFQDN"
  2. Change the code:

class GSSAPIMechanism(Mechanism): name = 'GSSAPI' score = 100 qops = QOP.all

allows_anonymous = False
uses_plaintext = False
active_safe = True

def __init__(self, sasl, principal=None, **props):
    Mechanism.__init__(self, sasl)
    self.user = None
    self._have_negotiated_details = False
    self.host = self.sasl.host
    self.service = self.sasl.service
    self.principal = principal
    self._fetch_properties('host', 'service')

    krb_service = '@'.join((self.service, self.host))         ##self.host here should be hostname rather than host ip

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/python-zk/kazoo/issues/636#issuecomment-765098707, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIFTHRBGNDVLZI724FP2M3S3DXH3ANCNFSM4WL3RDRA .

tianchao-haohan commented 3 years ago
tianchao-haohan commented 3 years ago

@ceache hi, is there any update on this issue?

ceache commented 3 years ago

There are a number of issues or at least confusing points of data about your setup. So I am going to go about how things should look like.

First it seems your client's Kerberos environment is set up properly: hiveuser@HADOOP.COM

There are, however, a couple of issues with the information you gave me about the server.

  • server hostname: hw-ker-node1

  • server ip: 10.8.8.6

  • Zk server Jaas config, if possibleClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=false useTicketCache=true debug=false; };

So, first, your server(s) should all have a keytab for zookeeper/$ HOSTNAME@HADOOP.COM . you can change the service from "zookeeper" to whatever you want but you must then use the service="yourservice" parameter in the sasl_options. It could also be in a separate Kerberos realm, but X-realm setup are beyond the scope of this issue.

Once you have the keytab at hand, you need a "Server" section in the Zookeeper server JAAS config. The "Client" section only pertains to the Java Zookeeper client, here you need to configure the Server component. Below is based on what we use in our integration suite:

Server {

com.sun.security.auth.module.Krb5LoginModule required debug=true isInitiator=false useKeyTab=true keyTab="/path/to/the/above/keytab/file" storeKey=true useTicketCache=false principal="zookeeper/yourhostname@HADOOP.COM"; };

Once Zookeeper is setup with this file, you should be able to connect from Kazoo, simply by passing hosts=[hostname], sasl_options={"mechanism"="GSSAPI, service="zookeeper"} to KazooClient.

As for your point about using hostname instead of host IP, this is actually the recommended way to connect to a Kerberized service. Kazoo will pass the hostname if it was provided. That being said, SASL and Kerberos do not really care, as long as your server has that principal in its keytab (e.g. "zookeeper/1.2.3.4@HADOOP.COM ").

I hope this helps.

-- Charles-Henri de Boysson