search

element-hq / dendrite

Dendrite is a second-generation Matrix homeserver written in Go!
https://element-hq.github.io/dendrite/
GNU Affero General Public License v3.0
11 stars 3 forks source link

Handle duplicate kafka/naffka messages #300

Closed matrixbot closed 2 weeks ago

matrixbot commented 2 weeks ago

This issue was originally created by @erikjohnston at https://github.com/matrix-org/dendrite/issues/300.

Since we only persist the partition offset after we've processed the message it's possible for dendrite services to receive the last message from kafka again if it died at the right/wrong point.

Currently e.g. federation sender doesn't handle this (causing #290)

Two options:

  1. Ensure we handle duplicate messages
  2. Persist the results of processing the messages atomically with the offsets
matrixbot commented 2 weeks ago

This comment was originally posted by @erikjohnston at https://github.com/matrix-org/dendrite/issues/300#issuecomment-335523635.

Persist the results of processing the messages atomically with the offsets

This is probably the better option, but its probably going to be quite faffy to actually code due to having to figure out how to atomically persist things without pushing processing into the database layer and using DB transactions

matrixbot commented 2 weeks ago

This comment was originally posted by @neilalexander at https://github.com/matrix-org/dendrite/issues/300#issuecomment-1092640378.

We're NATS now.