Keytab paths differ in client.properties and server.properties

[Fobhep]

For Confluent Enterprise Support customers, we would strongly advise you to open a Support ticket which will be addressed within your Support contract SLA at https://support.confluent.io

Describe the issue When using upgrade playbooks from 5.5.4 to 6.6.1 - admin-client call for detecting under replicated topics failes due to a wrong configure keytab path in the /etc/kafka/client.properties file.

To Reproduce Block from inventory for the brokers

kafka_broker:
  hosts:
    group0-broker1.hcloud-kafka-lab.test:
      ansible_host: broker1
      broker_id: 1
      kafka_broker_kerberos_keytab_path: ~/kerbldap/keytabs/kafka-broker1.keytab
      kafka_broker_kerberos_principal: kafka/group0-broker1.hcloud-kafka-lab.test@HCLOUD-KAFKA-LAB.TEST
      kafka_broker_custom_properties:
        broker.rack: rack-1
    group0-broker2.hcloud-kafka-lab.test:
      ansible_host: broker2
      broker_id: 2
      kafka_broker_kerberos_keytab_path: ~/kerbldap/keytabs/kafka-broker2.keytab
      kafka_broker_kerberos_principal: kafka/group0-broker2.hcloud-kafka-lab.test@HCLOUD-KAFKA-LAB.TEST
      kafka_broker_custom_properties:
        broker.rack: rack-2
    group0-broker3.hcloud-kafka-lab.test:
      ansible_host: broker3
      broker_id: 3
      kafka_broker_kerberos_keytab_path: ~/kerbldap/keytabs/kafka-broker3.keytab
      kafka_broker_kerberos_principal: kafka/group0-broker3.hcloud-kafka-lab.test@HCLOUD-KAFKA-LAB.TEST
      kafka_broker_custom_properties:
        broker.rack: rack-3

block from created server.properties file on broker3

listener.name.broker.ssl.truststore.location=/var/ssl/private/kafka_broker.truststore.jks
listener.name.broker.ssl.truststore.password=confluenttruststorepass
listener.name.broker.ssl.keystore.location=/var/ssl/private/kafka_broker.keystore.jks
listener.name.broker.ssl.keystore.password=confluentkeystorestorepass
listener.name.broker.ssl.key.password=confluentkeystorestorepass
listener.name.broker.sasl.enabled.mechanisms=GSSAPI
listener.name.broker.sasl.kerberos.service.name=kafka
listener.name.broker.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
  useKeyTab=true \
  storeKey=true \
  keyTab="/etc/security/keytabs/kafka-broker3.keytab" \
  principal="kafka/group0-broker3.hcloud-kafka-lab.test@HCLOUD-KAFKA-LAB.TEST";

part of /etc/client.properties file on broker3

# Maintained by Ansible
sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka_broker.keytab" principal="kafka/group0-broker3.hcloud-kafka-lab.test@HCLOUD-KAFKA-LAB.TEST";

As you can see the paths in the listener config and the client.properties file differ.

Expected behaviour The keyTab paths should be equivalent

Environment (please complete the following information):

• OS: RHEL 7.4
• CP-Ansible Branch: 6.6.1
• Ansible Version 2.9

I guess I might be doing something wrong here?

[domenicbove]

Oh wow I think at some point along the way the logic changed on what the filename for that keytab file would be. I think the simplest solution is to set this variable under each host:

kafka_broker_keytab_path: /etc/security/keytabs/kafka-broker3.keytab

This is the variable used in the client.properties file generation. Its where cp-ansible cp's keytabs to on the kafka_broker hosts.

[Fobhep]

Thanks - that's a quick and helpful answer. Maybe an idea for a general fix would be setting the default value in roles/confluent.variables/defaults/main.yml kind of like this (not tested!)

kafka_broker_keytab_path: "/etc/security/keytabs/{{ kafka_broker_kerberos_keytab_path | default("kafka_broker.keytab") | basename }}"

[domenicbove]

Ya, that could work.

I think the logic behind the change was I wanted to standardize the keytab filenames like the keystore filenames are.

[Fobhep]

I think standardization makes generally a lot of sense :)

[domenicbove]

Also check this issue out: https://github.com/confluentinc/cp-ansible/issues/588

Starting in 6.1.2 and 6.2.0 we will have all.yml upgrading for you and you wont need the upgrade playbooks. This would handle changes to the keytab filename (and things will be way more idempotent). Downgrades even work with this new approach which is pretty cool.

[Fobhep]

yes - I am aware of and looking forward to that change :) I did not know it will be in 6.1.2 already. When will 6.1.2 be released? Is upgrading from 5.5.4 to 6.1.2 also possible then?

[domenicbove]

Ya to be more clear, the change is merged into the 6.1.x branch and that branch will become 6.1.2-post when 6.1.2 comes out. 6.2.0 should be out very soon.

5.5.4 -> 6.1.1 should work today if you want to try it out with the 6.1.x branch

[utkarsh5474]

Hey, closing this since it seems from the conversation that the later patches of 6.1 has the fix.