Custom log directories for each of the services is not working

[greenhorntechie]

Hi, I am trying to setup a single node cluster using cp-ansbile scripts. In my inventory file, I have the below configuration:

---
all:
  vars:
    ansible_connection: ssh
    ansible_user: ubuntu
    ansible_become: true
    ansible_ssh_common_args: -o StrictHostKeyChecking=no

    # Custom log directory locations for each Confluent Kafka service
    kafka_broker_log_dir: "/confluent/kafka/logs"
    zookeeper_log_dir: "/confluent/zookeeper/logs"
    schema_registry_log_dir: "/confluent/schema-registry/logs"
    ksql_log_dir: "/confluent/ksql/logs"
    kafka_connect_log_dir: "/confluent/kafka-connect/logs"
    control_center_log_dir: "/confluent/control-center/logs"
    kafka_rest_log_dir: "/confluent/rest-proxy/logs"

    # All the custom properties for Zookeeper service
    zookeeper_custom_properties:
      dataDir: "/confluent/zookeeper/data"

    # All the custom properties for Kafka broker service
    kafka_broker_custom_properties:
      log.dirs: "/confluent/kafka/data"

Here, the directories /confluent/kafka/log, /confluent/kafka/data, /confluent/zookeeper/data and /confluent/zookeeper/logs are getting created as expected. However, all the other log directories for SR, C3 etc are not getting created under /confluentdirectory.

Also, when I check the Kafka server log, I am seeing the following error, post which the Kafka broker service stopped

[2022-04-20 18:59:48,446] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID mViGR6GIQSOvwPvv5iVrPQ doesn't match stored clusterId Some(ANqMwELUQv-SjgKuA7p5xA) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
        at kafka.server.KafkaServer.startup(KafkaServer.scala:342)
        at kafka.Kafka$.main(Kafka.scala:108)
        at kafka.Kafka.main(Kafka.scala)
[2022-04-20 18:59:48,456] INFO shutting down (kafka.server.KafkaServer)
[2022-04-20 18:59:48,466] INFO [feature-zk-node-event-process-thread]: Shutting down (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-04-20 18:59:48,472] INFO [feature-zk-node-event-process-thread]: Stopped (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-04-20 18:59:48,472] INFO [feature-zk-node-event-process-thread]: Shutdown completed (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-04-20 18:59:48,473] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2022-04-20 18:59:48,475] WARN An exception was thrown while closing send thread for session 0x10000ac467f0001. (org.apache.zookeeper.ClientCnxn)
EndOfStreamException: Unable to read additional data from server sessionid 0x10000ac467f0001, likely server has closed socket
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2022-04-20 18:59:48,577] INFO EventThread shut down for session: 0x10000ac467f0001 (org.apache.zookeeper.ClientCnxn)
[2022-04-20 18:59:48,577] INFO Session: 0x10000ac467f0001 closed (org.apache.zookeeper.ZooKeeper)
[2022-04-20 18:59:48,579] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2022-04-20 18:59:48,591] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2022-04-20 18:59:48,592] INFO shut down completed (kafka.server.KafkaServer)
[2022-04-20 18:59:48,592] ERROR Exiting Kafka. (kafka.Kafka$)
[2022-04-20 18:59:48,606] INFO shutting down (kafka.server.KafkaServer)

I am using the latest 7.1.0 version of the cp-ansible scripts to deploy the cluster services on a single node Ubuntu server running 20.04 version. The Ansible version used to run the scripts is 2.12.4

Seems like the services are not working as expected when the above configuration is used. Can you please fix and / or let me know the solution for this.

Thanks

[greenhorntechie]

Hi, I have cleaned up the log and data directories and then deployed the services once again. Since then, all the services have started and are running fine. However, the original issue i.e. custom log directories for various Kafka services still remains. Please advise whether this is a bug or whether I am doing something wrong here.

[greenhorntechie]

Hi, a quick update. After cleaning up all the data and log directories, I did a fresh installation using the below configuration. All the custom log directories are getting created this time. So I guess it was due to some stale data probably they were not created last time around.

---
all:
  vars:
    ansible_connection: ssh
    ansible_user: ubuntu
    ansible_become: true
    ansible_ssh_common_args: -o StrictHostKeyChecking=no

    # Custom log directory locations for each Confluent Kafka service
    kafka_broker_log_dir: "/confluent/kafka/logs"
    zookeeper_log_dir: "/confluent/zookeeper/logs"
    schema_registry_log_dir: "/confluent/schema-registry/logs"
    ksql_log_dir: "/confluent/ksql/logs"
    kafka_connect_log_dir: "/confluent/kafka-connect/logs"
    control_center_log_dir: "/confluent/control-center/logs"
    kafka_rest_log_dir: "/confluent/rest-proxy/logs"

    # All the custom properties for Zookeeper service
    zookeeper_custom_properties:
      dataDir: "/confluent/zookeeper/data"

    # All the custom properties for Kafka broker service
    kafka_broker_custom_properties:
      log.dirs: "/confluent/kafka/data"

As it was an issue with my deployment approach rather than the cp-ansible scripts, I am closing the issue as resolved. Thanks