PeerDB-io / peerdb

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
https://peerdb.io
Other
2.19k stars 90 forks source link

Stabilizing Query Based Replication #156

Closed saisrirampur closed 1 year ago

saisrirampur commented 1 year ago
  1. When source table in PG is empty - getting the below error:
    "message": "failed to get partitions from source: unsupported type: <nil>",
    "source": "GoSDK",
    "stackTrace": "",
    "encodedAttributes": null,
    "cause": {
    "message": "unsupported type: <nil>",
    "source": "GoSDK",
    "stackTrace": "",
    "encodedAttributes": null,
    "cause": null,
    "applicationFailureInfo": {
      "type": "",
      "nonRetryable": false,
      "details": null
    }
    },
    "applicationFailureInfo": {
    "type": "wrapError",
    "nonRetryable": false,
    "details": null
    }
    }
  2. In query based replication, we should allow 2 MIRRORs with same name. But as we removed the unique constraint for name (because of multi-table CDC), we are allowing. We might need to rethink the data-model for flows table. One approach is to separate metadata tables for CDC and Query Based Replication. image
iskakaushik commented 1 year ago

closing this after successful scale testing.