jduckwo
commented
3 weeks ago I am running into a similar issue. It appears that the subscriber to the replication slot is not being registered properly. When I check the active status of the replication slot, it is set to active=false and the restart_lsn is not advancing as expected. I have dropped/recreated the PG publication, PG Replication Slot, AB Source, AB Destination, and AB Connection objects and still cannot get the incremental replication to work properly. I also verified in the Airbyte logs that the query against the pg_replication_slots view does find the replication slot.
Here is my current config:
- Airbyte version: 1.1.0
- Airbyte Postgres Source Version: 3.6.22
- Airbyte Postgres Destination Version: 2.4.0
Postgres config:
- Postgres 16
ALTER SYSTEM SET wal_keep_size = '1GB';
SELECT pg_reload_conf();
SHOW wal_keep_size;
CREATE ROLE airbyte_role;
ALTER ROLE airbyte_role REPLICATION LOGIN;
GRANT SELECT ON TABLE public.table_1 TO airbyte_role;
ALTER TABLE public.table_1 REPLICA IDENTITY DEFAULT;
CREATE PUBLICATION airbyte_publication FOR TABLES IN SCHEMA public;
SELECT pg_create_logical_replication_slot('airbyte_slot', 'pgoutput');I also would appreciate any help getting this resolved. If helpful, I can provide my logs in addition to the ones in the original post.
Nanagoud007
marcosmarxm
rodireich
rowanmoul
Topic
I have enabled Change Data Capture (CDC) on my PostgreSQL source following the instructions provided in the Airbyte PostgreSQL source documentation. However, during each sync, Airbyte is loading the entire dataset instead of only the changes. This issue persists even when there are no updates in the source database.
Relevant information
Steps Taken:
Logs:This 4th time ran sync logdefault_workspace_logs_1388_txt.txt
Expected Behavior: Airbyte should only sync the changed records instead of reloading the entire dataset on every sync.
Actual Behavior: Each sync loads the whole dataset, resulting in increased load times and performance issues.
Additional Information: PostgreSQL Version: 16 Airbyte Version: 0.58.0 >
Questions:
I would appreciate any assistance in diagnosing and resolving this problem.