search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.41k stars 577 forks source link

pandaproxy: Create verifiable consumer and producer #9210

Open michael-redpanda opened 1 year ago

michael-redpanda commented 1 year ago

Develop a new service, somewhat akin to KgoVerifierService, that will exercise pandaproxy and verify that the same data that was produced was consumed.

JIRA Link: CORE-1192

NyaliaLui commented 1 year ago

Spoke offline with @michael-redpanda and @BenPope A verifiable consumer and producer is needed so we can test that the internal Kafka client works in scenarios where leadership changes while there are active producers and consumers. We do not need separate services to test this however, instead we could rely on the DT orchestration node to send produce/consume REST requests over a period of time.

With this in mind, we are changing this ticket to be the following:

Create a producer and consumer which send and fetch records over a period of time. Then create a test that will change leadership for topic-partitions while these entities are running. The producer and consumer should continue working even in the face of leadership changes.