search

risingwavelabs / risingwave

SQL stream processing, analytics, and management. We decouple storage and compute to offer instant failover, dynamic scaling, speedy bootstrapping, and efficient joins.
https://www.risingwave.com/slack
Apache License 2.0
6.59k stars 540 forks source link

Test more pg-types of postgres-cdc #16455

Open KeXiangWang opened 2 months ago

KeXiangWang commented 2 months ago

Like enum, some of the postgres rows cannot be directly parsed by String. We need more tests to verify different types. https://github.com/risingwavelabs/risingwave/pull/16423#discussion_r1575776831

StrikeW commented 2 months ago

More context: in cdc backfill we use Rust client to scan the upstream table, and we need to covert those rows in postgres format into our OwnedRow to store in our storage. For data types not supported in RW we will fallback to VARCHAR, which is supported in the DebeziumJson parser but need additional work to handle the format conversion in cdc backfill. Note that this requirement also apply to other cdc connector support cdc-backfill, e.g. mysql-cdc. cc @fuyufjh

StrikeW commented 2 months ago

We can create a cdc backfill table in https://github.com/risingwavelabs/risingwave/blob/adf13411bf652e30d62a08a7b0179de132611fe3/integration_tests/postgres-cdc/compatibility-rw.sql#L2 to cover the data type test.

KeXiangWang commented 2 months ago

We can create a cdc backfill table in

Okk I can add this.