message_middleware: make message passing from Replication.Server -> Replication.Publisher configurable

cpursley / walex

Postgres change events (CDC) in Elixir

MIT License

276 stars 14 forks source link

message_middleware: make message passing from Replication.Server -> Replication.Publisher configurable #65

Closed DaemonSnake closed 2 months ago

DaemonSnake commented 3 months ago

the aim of this is to allow the end user to implement a backpresure system if need be for instance using broadway / dets / persistent_ets / gen_stage ...

Taking broadway as example:

message_middleware would publish events directly to SQS or cast to an SQS publisher
the broadway consumer would receive the message and invoke WalEx.Replication.Publisher.process_message_sync himself

DaemonSnake commented 3 months ago

arf sorry was meant for internal PR, not ready yet 😓

cpursley commented 3 months ago

Let's reopen - maybe open a Discussion as well?