search

noctarius / timescaledb-event-streamer

timescaledb-event-streamer is a command line program to create a stream of CDC (Change Data Capture) TimescaleDB Hypertable events from a PostgreSQL installation running the TimescaleDB extension.
Apache License 2.0
40 stars 3 forks source link

NATS core support added #143

Open falkolab opened 10 months ago

falkolab commented 10 months ago
noctarius commented 10 months ago

I'd still be hesitant to merge this since non-Jetstream topics don't guarantee ordering, do they? 🤔

noctarius commented 10 months ago

Ok it seems ordering it available (https://docs.nats.io/reference/faq#does-nats-offer-any-guarantee-of-message-ordering) but the delivery guarantee for non-Jetstream is at-most-once which will still lead to issues (https://docs.nats.io/reference/faq#does-nats-guarantee-message-delivery).

falkolab commented 10 months ago

Yes, it's not stream but pub/sub and we have one publisher

Does NATS offer any guarantee of message ordering? NATS implements source ordered delivery per publisher. That is to say, messages from a given single publisher will be delivered to all eligible subscribers in the order in which they were originally published. There are no guarantees of message delivery order amongst multiple publishers.

falkolab commented 10 months ago

In some cases, the only fact of changes makes sense. It's up to the user to decide how to use it.

Yes, for Core pub/sub, it's at-most-once delivery. If there are no subscribers on the topic, the message will not be delivered and will be skipped. For Openfaas CE, I can't use jetstream because it's a Pro and paid functionality. My function reacts to CDC and requests initial data on start, then tracks the CDC events.

noctarius commented 9 months ago

Hey @falkolab! Sorry for the long wait.

I'm still hesitant to put this into the core.

I'm thinking about adding plugins to the distribution though. Meaning, it'd be delivered as part of the download, but needs to be specifically enabled. However, that would work on Linux only, due to the plugin system in Go.

Any reason, you can't use it as a plugin?

falkolab commented 9 months ago

Hey @falkolab! Sorry for the long wait.

I'm still hesitant to put this into the core.

I'm thinking about adding plugins to the distribution though. Meaning, it'd be delivered as part of the download, but needs to be specifically enabled. However, that would work on Linux only, due to the plugin system in Go.

Any reason, you can't use it as a plugin?

Yes I can. I can even use my forked repository. If one need this functionality then welcome to my fork.

noctarius commented 9 months ago

Looking into this again. How would you write an actual test for this sink when you it's at-most-once delivery. While it will work in 99% of the cases, the 1% spurious failure would be bothersome 🤔

My current idea would be to mark sinks are "potentially complicated" (whatever the final term would be), meaning you should only use them when you know what you do. That would generate a warning during startup and may need to be enabled specifically.

Anyhow, it would still require a clean integration test.