Question: Is it at-most-once delivery semantic?

ihippik / wal-listener

PostgreSQL WAL listener

Apache License 2.0

296 stars 44 forks source link

Question: Is it at-most-once delivery semantic? #21

Closed Hixon10 closed 6 months ago

Hixon10 commented 7 months ago

Hi there,

I want to confirm my understanding of this library. In case, when we read a new event from PostgreSQL replication log, but isn't able to write it to Kafka/NATS, we are going to lose this data. Hence, it is at-most-once delivery semantic. Is it correct, or am I wrong?

Hixon10 commented 7 months ago

It looks like that I am wrong, based on - https://github.com/ihippik/wal-listener/blob/ef2311852ef336250bad81630450d573eca33783/listener/listener.go#L234-L256

However, I still wonder, are there any well-known use-cases, when we can lose data.

ihippik commented 7 months ago

Sorry for delay, I'll do some research on this. but in any case I should add some metrics with potential data problems

ihippik commented 6 months ago

so, I added straightforward behavior - an error occurred while processing a message, then we forward it to the top and restart the service (without ack). And in this case we will get it again (at least once). I also added a metric by which you can evaluate the number of problems.