search

dudash / openshift-microservices

:memo: Example showcasing how to leverage OpenShift and it's runtimes to do microservices
Apache License 2.0
9 stars 21 forks source link

Why Kafka? And how would that work on OpenShift? #4

Open dudash opened 5 years ago

dudash commented 5 years ago

A question I've gotten informally when discussing this with colleague was "Why use Kafka and how does that run on OpenShift/Kubernetes?"

Specifically, the "Why Kafka" was targeted at other options for microservices to communicate with each other (like REST, gRPC, etc.). This is probably the more important question of the two for this repo.

The "How does it work in K8s/OpenShift?" was looking to understand how a multi-broker with zookeeper would work in a managed container platform like OpenShift.

So I'm starting that discussion here.

dudash commented 5 years ago

How would it work on OpenShift is straightforward to answer. And I'll punt to someone else's documentation. See the following articles:

  1. https://developers.redhat.com/blog/2018/10/25/welcome-apache-kafka-to-the-kubernetes-era/
  2. https://developers.redhat.com/blog/2018/10/29/how-to-run-kafka-on-openshift-the-enterprise-kubernetes-with-amq-streams/
dudash commented 5 years ago

For reference the standard Kafka use cases: https://kafka.apache.org/uses

In this case Kafka is only being used for a subset of the communications of the services. So REST/gRPC aren't off the table. We are streaming data from multiple decoupled services into Kafka for the purposes of a search service. It's similar to the discussion on this stackexchange:

Because this is just a demo repo with no actual users or requirements to meet, this seemed like an interesting example to showcase as part of the architecture.

dudash commented 5 years ago

an alternative (not chosen) for the search service to get access to multiple service's data would be to use a composition pattern like this: image

dudash commented 4 years ago

This might need to be reconsidered now that istio/jaeger should be able to leverage Kafka (https://github.com/envoyproxy/envoy/pull/8188) and we are considering making istio first-class in this demo (#43 )