implement channel for kafka - Githubissues

invesdwin / invesdwin-context-integration

invesdwin-context modules that provide integration features

GNU Lesser General Public License v3.0

1 stars 0 forks source link

implement channel for kafka #14

Open subes opened 3 years ago

subes commented 3 years ago

https://kafka.apache.org/

subes commented 2 months ago

We can use https://testcontainers.com/ to create an isolated repeatable performance test case. We can use an existing docker container and parameterize it (https://java.testcontainers.org/modules/kafka/), or we can modify the docker file directly to create the scenario we need (https://docs.docker.com/ https://hub.docker.com/r/apache/kafka). A similar example exists for hadoop/yarn and MpjExpress here: https://github.com/invesdwin/invesdwin-context-integration/blob/main/invesdwin-context-integration-parent/invesdwin-context-integration-mpi-test/src/test/java/de/invesdwin/context/integration/mpi/test/MpjExpressYarnTest.java

We can integrate the kafka streams api as a channel endpoint reader/writer. Examples:

We should test the performance of:

transient topics
persistent topics
different settings of those (compression, durability, availability, etc)
different topologies (multiple containers): https://github.com/wurstmeister/kafka-docker https://java.testcontainers.org/modules/docker_compose/
compare kafka-native (https://hub.docker.com/r/apache/kafka-native/) against kafka (https://hub.docker.com/r/apache/kafka/)

We can then put those performance results into the https://github.com/invesdwin/invesdwin-context-integration/blob/main/README.md#synchronous-channels and maybe continue by:

comparing it against a direct competitor like pulsar: https://github.com/invesdwin/invesdwin-context-integration/issues/15
extending the testsuite for throughput tests in addition to the existing ping-pong latency tests
replicating the setup on actual hardware (maybe even specialized low-latency network cards)