Backend example not working with mojaloop - kafka init containers not finishing

[mihai-tarabuta-yt]

Hi there,

Followed the instructions here https://docs.mojaloop.io/technical/deployment-guide/ to deploy mojaloop (v15.0.0) on Azure AKS. It does not work out of the box and the problem appears to be the kafka init containers not being able to connect to kafka controllers. kafka init container logs: Waiting for Kafka... [2024-04-29 23:25:54,518] WARN [LegacyAdminClient clientId=admin-1] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) Exception in thread "main" java.lang.RuntimeException: Request METADATA failed on brokers List(kafka:9092 (id: -1 rack: null)) at kafka.admin.BrokerApiVersionsCommand$AdminClient.sendAnyNode(BrokerApiVersionsCommand.scala:159) at kafka.admin.BrokerApiVersionsCommand$AdminClient.findAllBrokers(BrokerApiVersionsCommand.scala:182) at kafka.admin.BrokerApiVersionsCommand$AdminClient.awaitBrokers(BrokerApiVersionsCommand.scala:174) at kafka.admin.BrokerApiVersionsCommand$.execute(BrokerApiVersionsCommand.scala:61) at kafka.admin.BrokerApiVersionsCommand$.main(BrokerApiVersionsCommand.scala:55) at kafka.admin.BrokerApiVersionsCommand.main(BrokerApiVersionsCommand.scala)

Kafka Controller logs [2024-04-29 23:37:29,574] INFO [SocketServer listenerType=BROKER, nodeId=0] Failed authentication with /10.244.0.185 (channelId=10.244.0.227:9092-10.244.0.185:45134-41935) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector) ... [2024-04-29 23:37:49,862] INFO [SocketServer listenerType=BROKER, nodeId=0] Failed authentication with /10.244.0.185 (channelId=10.244.0.227:9092-10.244.0.185:51610-41937) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector) When this happens the containers "wait-for-kafka" will chew up all the CPU resources and may give the wrong impression that the cluster/node is saturated. tried with a different version of contaner init (solsson/kafka:3.3.0 instead of solsson/kafka:2.8.1) but the same problem exists.

[devzaidi]

I am also running into this

[kiswend]

Hi @devzaidi , are you running on Azure too?

[kiswend]

Hi @mihai-tarabuta-yt , we will more details of your implementation. which version of k8s are you using? Are you using a customer values files?

[devzaidi]

Hi I am attempting this on Docker Desktop on an M3 Pro Macbook. My kubernetes version is v1.29.2. I am using the latest helm charts for Mojaloop which I assume point to v16.0. I have not made any changes to the values other than change mysql 5.7 to 8.0 for the initContainers.

[mihai-tarabuta-yt]

Hi kiswend, We have AKS v 1.28.5 We did not use a customer values file - just used the instructions provided on the mojaloop site. Happy to use a customized values file if you can advice what to change. At the moment, the "wait-for-kafka" init containers are stuck when executing this command:

sh -c until ./bin/kafka-broker-api-versions.sh --bootstrap-server ${KAFKA_HOST}:${KAFKA_PORT}; do echo --------------------; echo Waiting for Kafka...; sleep 2; done; echo ====================; echo Kafka ok!;

This causes the error I mentioned

[akihisa-wiredin]

@mihai-tarabuta-yt @devzaidi I experienced the same issue. Please try the following settings in your custom values file.

kafka:
  listeners:
    client:
      protocol: PLAINTEXT

This is intended for development environments, and for production environments, a more secure configuration might be necessary.