toluaina
commented
10 months ago My guess is this has to do with AWS Aurora. PGSync has customs modifications for AWS RDS.
can you please run this query on your db
SELECT name FROM pg_settings ORDER BY name
PGSync version: 2.5.0
Postgres version: Aurora 14.5
Elasticsearch version: Opensearch 2.3
Redis version: 5.0.7
Python version: 3.8.10
Problem Description: If i run pgsync everything works, i get the data inserted into opensearch, but if i try to run it using the --daemon flag, pgsync detect the changes on the rds, but it never insert them into opensearch
the only thing i see is:
Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]... Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]... Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]... Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]...
i tried deleting the construct file and start the process again, still same issue.
i also followed this steps:
Did you run bootstrap successfully? did you alter the structure of the schema.json There is a checkpoint file which maintains the state. Try deleting .construct_pg_posts and rerun
if i restart pgsync on deamon mode, it inserts the rows that were pending from the previous run, but then any new row is not inserted.
the schema definition im using is:
[
{ "database": "client_facing", "index": "blue-sky-rules-schema", "nodes": { "table": "v_security_stats", "schema": "ref_data", "columns": [ "source", "status", "secid", "comp_id", "cusip", "issuer_name", "symbol" ], "primary_key": ["secid"] } } ]
Error Message (if any):