Blumchen - Push-based Outbox for PostgreSQL

Outbox Pattern with CDC and .NET based on Postgres logical replication with Npgsql integration.

Read more details in:

• Push-based Outbox Pattern with Postgres Logical Replication.
• How to get all messages through Postgres logical replication.

Features:

• Publication filter is enabled to trigger only INSERTed rows;
• AOT compliant compilation enforced by design

Main logic is placed in EventsSubscription.

Running source code locally

1. Start Postgres with WAL enabled from Docker image.

   docker-compose up

2. Run(order doesn't matter) Publisher and Subscriber apps, under 'demo' folder, from vs-studio, and follow Publisher instructions.

Testing (against default docker instance)

Run tests

dotnet test

Links

Postgres Logical Replication

• Postgres Docs - Postgres logical replication
• Npgsql - Logical Replication
• Robert Treat - Logical Replication...LIVE!
• Dmitry Narizhnykh - PostgreSQL Change Data Capture and Golang Sample Code

WAL

• Devrim Gündüz -WAL: Everything you want to know
• Postgres Documentation - Write Ahead Log
• The Internals of PostgreSQL - Write Ahead Logging — WAL
• Hevo - Working With Postgres WAL Made Easy

Logical Replication

General Introduction

• Gunnar Morling - The Wonders of Postgres Logical Decoding Messages
• Gunnar Morling - Open-source Change Data Capture With Debezium - video
• Gunnar Morling - Open-source Change Data Capture With Debezium - slides
• Several9s - Using PostgreSQL Logical Replication to Maintain an Always Up-to-Date Read/Write TEST Server
• Matt Tanner - PostgreSQL CDC: A Comprehensive Guide

Other

• Dmitry Narizhnykh - PostgreSQL Change Data Capture and Golang Sample Code
• Fujistsu - How PostgreSQL 15 improved communication in logical replication
• Kinsta - PostgreSQL Replication: A Comprehensive Guide
• Amit Kapila - Replication Improvements In PostgreSQL-14
• Postgresql Wiki - Logical Decoding Plugins
• Npgsql - Logical Replication
• Konstantin Evteev - Recovery use cases for Logical Replication in PostgreSQL 10
• EDB - PostgreSQL Write-Ahead Logging (WAL) Trade-offs: Bounded vs. Archived vs. Replication Slots
• Percona - The 1-2-3 for PostgreSQL Logical Replication Using an RDS Snapshot
• 2nd Quadrant - Basics of Tuning Checkpoints
• Thiago - How to use Change Data Capture (CDC) with Postgres
• Ramesh naik E - Change Data Capture(CDC) in PostgreSQL
• Wal2Json - JSON output plugin for changeset extraction
• AWS Database Blog - Using logical replication to replicate managed Amazon RDS for PostgreSQL and Amazon Aurora to self-managed PostgreSQL
• AWS Database Blog - Stream changes from Amazon RDS for PostgreSQL using Amazon Kinesis Data Streams and AWS Lambda
• PGDeltaStream - Streaming Postgres logical replication changes atleast-once over websockets
• Hrvoje Milković - Replicate PostreSQL data to Elasticsearch via Logical replication slots
• Azure - Azure Database for PostgreSQL : Logical Replication
• Azure Database for PostgreSQL—Logical decoding and wal2json for change data capture
• Amit Langote - Postgresql To Kinesis For Java - Disney Streaming

Queue

• Adriano Caloiaro - Choose Postgres queue technology

Performance

• 2ndQuadrant - Performance limits of logical replication solutions
• Some benchmarking techniques

Snapshots

• Christos Christoudias - Creating a Logical Replica from a Snapshot in RDS Postgres

Ordering

• Virender Singla - Postgres — Logical Replication and long running transactions
• Ordering with Advisory Locks

Partitioning

• Postgres: partitioned tables can now be replicated

Locks

• Depesz - Picking task from queue – revisit
• Alvaro Herrera - Waiting for 9.5 – Implement SKIP LOCKED for row-level locks
• Chris Hanks - Turning PostgreSQL into a queue serving 10,000 jobs per second
• Vlad Mihalcea - How do PostgreSQL advisory locks work
• Marco Slot - When Postgres blocks: 7 tips for dealing with locks
• Marco Slot - PostgreSQL rocks, except when it blocks: Understanding locks
• Nickolay Ihalainen - PostgreSQL locking, Part 1: Row Locks
• Nickolay Ihalainen - PostgreSQL locking, part 2: heavyweight locks
• Nickolay Ihalainen - PostgreSQL locking, part 3: lightweight locks

Outbox implementations

• Robert Kawecki - esdf2-eventstore-pg