search

terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

Dev Tooling: Add support for kafka and minio tools/brokers in k8s and e2e #3759

Closed sotojn closed 2 weeks ago

sotojn commented 2 months ago

This issue branches out into two similar issues:

  1. As of right now, we have no way of accessing a kafka broker outside of the dev environment. That means we can only use kcat to list brokers and we are unable to side cart a kafka UI outside of the cluster to monitor kafka topics, messages, consumers, and consumer offsets.
  2. We also can't access the minio UI because it points to a random port that we don't have access to. The minio UI has a lot of useful tools that can be used for debugging such as trace log monitoring, information on bucket naming rules, etc.

Having easy access to these things will help give us more insight on related failing tests in e2e and give us more information on these services when running a local dev cluster.