InternetHealthReport / kafka-toolbox

Collection of producers, consumers, and basic tools for IHR kafka cluster
GNU General Public License v3.0
4 stars 9 forks source link

Create docker container environment for Kafka-Toolbox repo #15

Open ritikBhandari opened 1 year ago

ritikBhandari commented 1 year ago

@romain-fontugne I was planning to create an entire docker environment for our kafka server. But what I have observed is we have around 6-7 producers and a bunch of consumers in psql/consumers. Do you have any existing approach as to how to club these producers and consumers? I have 2 options:

  1. Creating separate containers for each producer and consumer which will allow us to scale specific components if they generate data with high velocity and have fine-grained control over the components. But this can create a little bit of chaos as well since we'll have to deploy 20+ containers.

  2. If we want a simpler deployment and management experience, we could combine all of the producers into one container and all of the consumer into another container. Just the fact I feel is it would limit our ability to scale individual components independently.

But entirely your call. We can move forward accordingly.

romain-fontugne commented 1 year ago

I think the first approach would be better, especially when things goes wrong we can restart only the failing producer/consumer

ritikBhandari commented 1 year ago

Makes sense!