search

tvkitchen / countertop

The entry point for developers who want to set up a TV Kitchen.
https://tv.kitchen
GNU Lesser General Public License v3.0
6 stars 2 forks source link

We need to sort out kafka #69

Open slifty opened 4 years ago

slifty commented 4 years ago

Task

Description

We're using Kafka for managing our data streams, but right now we're just using whatever comes out of the box from kafkajs, without much clarity on exactly how it works and how we interact with it.

Specifically:

  1. Data lives in kafka for a while; we probably don't want very long retention policies.
  2. When a new TV Kitchen worker connects, we don't want it to read historic messages (I believe that kafkaJS has a fromBeginning that probably defaults to false, but we should confirm).
  3. Are we supposed to be tagging messages in any special way?

I don't know what code will come out of this issue, but I suspect we have some configurations to make (at the very least to our kafka service + documentation).

Relevant Resources / Research

Related Issues

This will likely impact how we interact with kafka in the countertop, so #23 is a related issue.

slifty commented 4 years ago

Another example of a problem: when I start / restart during testing it seems that consumers are not automatically de-registering after dying. This means that messages are being sent to the now-dead consumers instead of the active consumer, so the active consumer just sits there silently.

chriszs commented 4 years ago

I set KAFKA_LOG_RETENTION_BYTES in the Docker Compose just now as test to see whether that improved Kafka performance. Seems like what we may want here. The alternative is setting retention by hours.