IBM / sarama

Sarama is a Go library for Apache Kafka.
MIT License
11.58k stars 1.76k forks source link

Integrated Performance Testing Tool for Producer-Consumer Workflow #3007

Open adil-faiyaz98 opened 3 weeks ago

adil-faiyaz98 commented 3 weeks ago

Description : Introduce a performance-oriented testing tool in the library to measure the end-to-end performance of producer-consumer workflows, specifically focusing on how efficiently the message broker accepts and distributes messages, and how quickly the consumer application processes and persists these messages into the database

Problem Statement: In a message-driven architecture, ensuring optimal performance from message production to final data persistence is crucial. Currently, there is a lack of tools to assess the entire performance chain from the message producer through the broker to the consumer application and database. Although individual components like the broker or database might have their own performance benchmarks, understanding the integrated performance across these systems is essential for identifying potential bottlenecks and achieving robust, real-time data processing.