canonical / dqlite

Embeddable, replicated and fault-tolerant SQL engine.
https://dqlite.io
Other
3.77k stars 213 forks source link

Method to be informed about latest change in table #551

Closed koh-osug closed 5 months ago

koh-osug commented 5 months ago

I have business logic which needs to be notified as soon as data is added, changed or deleted. Are there any approaches available how to do this?

I could restructure my table layout and add a column “deleted” and use a creation and modification date for these purposes, but maybe there is already a listener concept available.

I addition I have find a way how to detect the latest changes, i.e. I cannot iterating over the full database but have to look for changes since the last check. Is it safe to use for this last check a timestamp? I assume not all clocks of all nodes are synchronized, but if I take the granularity of one minute, is it a good idea to query then “all rows deleted after”, “all rows modified after”, "“all rows created after”?

PS: I have also added this to discourse, but since I'm the only uncategorized message there it might be not the right place.

cole-miller commented 5 months ago

I could restructure my table layout and add a column “deleted” and use a creation and modification date for these purposes, but maybe there is already a listener concept available.

There is not, unfortunately.

I addition I have find a way how to detect the latest changes, i.e. I cannot iterating over the full database but have to look for changes since the last check. Is it safe to use for this last check a timestamp?

All insertions are executed on the current leader and then replicated exactly to other nodes, so for as long as the leader remains stable the timestamps will not jitter any more than the clock used by SQLite. When the leader changes you could see a discontinuity in the timestamps.

As an alternative to timestamps, have you considered just using a monotonically increasing integer key? This should be straightforward to do using SQL and doesn't have the consistency problem of timestamps.

koh-osug commented 5 months ago

The auto-increment is a nice idea. Thanks.