invesdwin / invesdwin-context-integration

invesdwin-context modules that provide integration features
GNU Lesser General Public License v3.0
1 stars 0 forks source link

implement channel for kafka #14

Open subes opened 3 years ago

subes commented 3 years ago

https://kafka.apache.org/

subes commented 1 month ago

We can use https://testcontainers.com/ to create an isolated repeatable performance test case. We can use an existing docker container and parameterize it (https://java.testcontainers.org/modules/kafka/), or we can modify the docker file directly to create the scenario we need (https://docs.docker.com/ https://hub.docker.com/r/apache/kafka). A similar example exists for hadoop/yarn and MpjExpress here: https://github.com/invesdwin/invesdwin-context-integration/blob/main/invesdwin-context-integration-parent/invesdwin-context-integration-mpi-test/src/test/java/de/invesdwin/context/integration/mpi/test/MpjExpressYarnTest.java

We can integrate the kafka streams api as a channel endpoint reader/writer. Examples:

We should test the performance of:

We can then put those performance results into the https://github.com/invesdwin/invesdwin-context-integration/blob/main/README.md#synchronous-channels and maybe continue by: