search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.71k stars 591 forks source link

compatibility: meta-issue for phase 2 of compat testing #2928

Open NyaliaLui opened 3 years ago

NyaliaLui commented 3 years ago

See Phase 1 at #2160

Motivation

A primary goal of Redpanda is to provide “Kafka compatibility” which means that applications that work with Kafka should also work unmodified with Redpanda. Achieving Kafka compatibility is difficult because there is no formal specification that describes how a library can ensure it works with all of Kafka. At Vectorized, we heavily rely on testing and customer feedback to reveal new incompatibilities. Instead, Vectorized should preemptively test Redpanda in ways that are likely to reveal incompatibilities so they may be addressed before customers encounter them.

Goal

Reach Kafka compatibility by testing various Kafka clients against Redpanda. A Kafka client is any API or application developed for Kafka. The results of this work will be a set of tests for each client as well as a script that calculates changes from the last successful test run. The script should also calculate Redpanda code coverage. The idea is to test for Kafka compatibility and see what Redpanda code clients use.

Kafka clients to test

This list may change as clients are evaluated and code coverage is improved. For example, some clients are developed with others such as kcat which uses librdkafka. Another example is if two clients have the same code coverage, then perhaps create a single test.

This meta-issue lists the Kafka clients to test in phase 2. Phase 1 is at #2160.

Kafka clients were chosen from various web searches and Apache's list of clients. Selection was based on two criteria:

  1. The number of stars on github -- Select those clients with higher star counts because they are likely to be used in Kafka applications and thus used against Redpanda
  2. The last date & time the client was updated -- Select those clients with latest updates because many clients are not up-to-date with KIPs or Kafka versions

Phase 2

JIRA Link: CORE-784

emaxerrno commented 3 years ago

@NyaliaLui are -all- of the Kafka streams examples that make sense already in the tree? I only saw a couple

NyaliaLui commented 3 years ago

@NyaliaLui are -all- of the Kafka streams examples that make sense already in the tree? I only saw a couple

@senior7515 9/16 examples are in the tree. The examples that aren't in the tree are:

emaxerrno commented 3 years ago

@NyaliaLui are -all- of the Kafka streams examples that make sense already in the tree? I only saw a couple

@senior7515 9/16 examples are in the tree. The examples that aren't in the tree are:

Thanks for the updates.

  1. outside of ducktape, do all of these pass?
  2. can we write a test suite outside of ducktape, say in bash, that actually accomplishes these tests, then we can partner w/ @dotnwat or someone to help you teach ducktape new tricks to get them to run there.
  3. i think the point is to fix these issues tho, so I'd say before moving to spark, we should probably fix all the redpanda issues
NyaliaLui commented 3 years ago

outside of ducktape, do all of these pass?

Yes. They all work outside of ducktape.

can we write a test suite outside of ducktape, say in bash, that actually accomplishes these tests, then we can partner w/ @dotnwat or someone to help you teach ducktape new tricks to get them to run there.

Sure I can write a test suite outside of ducktape.

i think the point is to fix these issues tho, so I'd say before moving to spark, we should probably fix all the redpanda issues

Sure, I'll get the issues fixed before moving on.

mensfeld commented 10 months ago

Achieving Kafka compatibility is difficult because there is no formal specification that describes how a library can ensure it works with all of Kafka.

My work (karafka + UI, waterdrop and rdkafka-ruby) have heavy integration suite that covers all the use-cases I could come up with. I do plan to roll out redpanda test suite so I do not thing you guys have to do it ;)