We've had the logic for advancing the main replication slot for some time now. That slot is used as a way to retain WAL records, allowing the sync service to maintain a window within which it can go back in time and replay old transactions.
A separate replication slot is used for the logical replication connection that the sync service maintains with Postgres when it's running. This slot is also known as the active replication slot.
The issue this PR addresses is caused by the fact that the active replication slot is never advanced when electrified tables aren't seeing any writes. This manifests as an endlessly growing replication lag and disk usage when the database sees a constant rate of writes to non-electrified tables:
With the fix in this PR in place, the sync service can now maintain constant replication lag that we can further adjust, for example, to prevent DigitalOcean from raising alerts about high replication lag. On the chart below you can see how disk usage goes down when the sync service is upgraded to a patched version: it never goes up the same way it used to:
And here's zoomed-in view of a 6-hour time span with the patched version running:
We've had the logic for advancing the main replication slot for some time now. That slot is used as a way to retain WAL records, allowing the sync service to maintain a window within which it can go back in time and replay old transactions.
A separate replication slot is used for the logical replication connection that the sync service maintains with Postgres when it's running. This slot is also known as the active replication slot.
The issue this PR addresses is caused by the fact that the active replication slot is never advanced when electrified tables aren't seeing any writes. This manifests as an endlessly growing replication lag and disk usage when the database sees a constant rate of writes to non-electrified tables:
With the fix in this PR in place, the sync service can now maintain constant replication lag that we can further adjust, for example, to prevent DigitalOcean from raising alerts about high replication lag. On the chart below you can see how disk usage goes down when the sync service is upgraded to a patched version: it never goes up the same way it used to:
And here's zoomed-in view of a 6-hour time span with the patched version running:
Fixes #1285.