Microservices and Kafka

[nehanims]

Modlarize app logic for better separation of concern

[nehanims]

Factors to consider in both Event Driven and Request Resonse architecture:

• Reactivity
• Coupling in Space and Time
• Consistency
• Historical State
• Architectural Flexibility
• Data Access and Data Reuse

https://www.youtube.com/watch?v=7fkS-18KBlw

Explore Kafka streams: https://www.youtube.com/watch?v=dsK-zd8pN_Q

[nehanims]

Four types of EDA:

• Application Internal
• Ephemeral Messaging
• Queues
• Publish/Subscribe https://www.youtube.com/watch?v=J-kKR3omk-g

[nehanims]

Designing event driven system: https://developer.confluent.io/courses/event-design/intro/

[nehanims]

https://youtube.com/playlist?list=PLa7VYi0yPIH145SVtPoh3Efv8xZ1ehUYy&si=L1NkFB4AB2NEJUFe

[nehanims]

Kafka is fast because of the following 2 factors:

• Write is fast because of sequential disk access since it is an append only log. What slows down magnetic disc is the physical movement of disk head during random access operation. So, even though it is persistent, it is a fast messaging system which also offers a flexible retention policy. It is also pretty cheap since sequential data access is fast on HDD which is substantially cheaper than SSD.
• Read is fast due to zero copy read, since kafka uses DMA to copy the data from OS buffer directly to the NIC (instead of the usual path where data would first be copied from the OS buffer -> kafka application buffer -> websocket buffer ->NIC buffer).

https://youtu.be/UNUz1-msbOM?si=BFHknyNEcmscb7e9

[nehanims]

Setting up a Kafka cluster using KRaft mode locally for your application is a great choice for simplifying deployment since KRaft eliminates the need for ZooKeeper, making Kafka clusters easier to manage. Below is a step-by-step guide to configuring a Kafka cluster using KRaft mode locally:

Step 1: Install Java

Kafka requires Java to run. Make sure you have Java installed:

java -version

If Java is not installed, install the latest version of the JDK.

Step 2: Download and Install Kafka

1. Download Kafka:

   • Visit the Apache Kafka download page and download the latest binary for your platform.

   • Alternatively, use wget to download Kafka directly:

     wget https://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz

2. Extract the downloaded file:

   tar -xzf kafka_2.13-3.6.0.tgz
   cd kafka_2.13-3.6.0

Step 3: Configure Kafka in KRaft Mode

1. Create a Data Directory:

   • Kafka stores its logs and metadata in a data directory. Create this directory:

     mkdir -p /tmp/kraft-combined-logs

2. Configure server.properties:

   • Kafka includes a sample configuration file, server.properties. Open it and make the following changes for KRaft mode:

     nano config/kraft/server.properties

   • Add or modify the following properties:

     # Cluster ID (You can generate one with the UUID command below)
     process.roles=broker,controller
     node.id=1
     controller.quorum.voters=1@localhost:9093
     listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
     log.dirs=/tmp/kraft-combined-logs
     # Generate a new Cluster ID

   • Generate a Cluster ID:

     KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

   • Format the storage directory:

     bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

Step 4: Start the Kafka Broker in KRaft Mode

1. Start the Kafka Broker:

   bin/kafka-server-start.sh config/kraft/server.properties

   • This starts the Kafka broker in KRaft mode, which acts as both the broker and controller.

Step 5: Verify Kafka is Running

1. Create a Topic:

   • In a new terminal, navigate to the Kafka directory and create a new topic:

     bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

2. List Topics:

   • Verify that the topic was created:

     bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Step 6: Install and Run Kafka Clients

1. Install Kafka CLI tools (optional):

   • If you prefer a GUI to interact with Kafka, you can use free tools like Kafdrop or Kafka Tool.
   • However, Kafka’s command-line tools (kafka-console-producer.sh and kafka-console-consumer.sh) are sufficient for testing.

2. Run a Producer:

   • Open a new terminal and run a producer:

     bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

   • Type some messages and hit Enter.

3. Run a Consumer:

   • In another terminal, run a consumer:

     bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

   • You should see the messages produced earlier.

Step 7: Configure Your Application Services

1. Kotlin Spring Boot Service:

   • Configure the Spring Kafka properties in your application.yml or application.properties:

     spring:
     kafka:
      bootstrap-servers: localhost:9092
      producer:
        key-serializer: org.apache.kafka.common.serialization.StringSerializer
        value-serializer: org.apache.kafka.common.serialization.StringSerializer

2. Python Consumer Service:

   • Install the Kafka Python library:

     pip install kafka-python

   • Use it in your Python code to consume messages from Kafka:

     from kafka import KafkaConsumer
     
     consumer = KafkaConsumer('test-topic', bootstrap_servers=['localhost:9092'])
     for message in consumer:
      print(f"Received message: {message.value.decode('utf-8')}")

Step 8: Test the Complete Workflow

1. Simulate Uploads:

   • Trigger the upload process from your frontend or a simple script that sends audio files to the Kotlin service.

2. Check Kafka Topics:

   • Ensure that your services are producing and consuming messages as expected.

3. Validate Transcriptions and Metadata:

   • Verify that the transcriptions and metadata are stored correctly in MinIO and your database.

Step 9: Monitor and Troubleshoot

1. Log Monitoring:

   • Keep an eye on the Kafka logs for any issues:

     tail -f logs/kafkaServer.out

2. Performance Tuning:

   • For large workloads, consider tuning Kafka and JVM parameters to optimize performance.

Conclusion

You now have a Kafka cluster running in KRaft mode locally, integrated with your application services for handling audio uploads, transcription, and metadata processing. This setup is fully open-source and should be sufficient for development and testing purposes.

[nehanims]

https://medium.com/@katyagorshkova/docker-compose-for-running-kafka-in-kraft-mode-20c535c48b1a

[nehanims]

use the topics in this article to ask questions https://www.udemy.com/course/spring-kafka-reactive/

[nehanims]

https://dev.to/tutorialq/building-scalable-applications-with-kafka-and-reactive-programming-5cea

[nehanims]

CHATGPT Discussion about :

• how to setup kafka in kraft mode
• pros and cons of using official kafka image vs either bitnami or confluent docker images
• how docker network can be setup to run one or more
• overview of differences between API gateway, Kubernetes and docker network capabilities and confirmation that they can all be layered together to leverage their individual functionalities, so it is a good idea to start out by creating everything (object store, kafka clusters, DB) as containers running on the same docker network and modify it when needed

[nehanims]

Kafka with Spring Boot

https://www.baeldung.com/kotlin/apache-kafka