sequinstream / sequin

Postgres CDC to streams and queues like Kafka, SQS, HTTP endpoints, and more
https://sequinstream.com
MIT License
493 stars 16 forks source link

Make Kafka's message `key` first-class #646

Closed acco closed 8 hours ago

acco commented 1 day ago

Like SQN-3414, we want to hoist up the concept of message key / blend with our concept of grouping. So that messages with the same group are routed to same partition.

RTLS commented 8 hours ago

One path forward is to extend group_id from ConsumerRecrod to ConsumerEvent - in this case, we can then simply use the group_id as the message_key.

Alternatively, we can pass in the full PostgresDatabaseTable and SequenceFilter then dynamically generate the message_key from the sequence_filter.group_column_attnums and the values in the record or event data.

This latter path requires fewer changes and also allows the message key to be changed dynamically, ie. without migrating values in the consumer records or consumer events tables.