Closed tchivs closed 1 week ago
@tchivs Thanks for you raising this issue. This seems to be an issue with flink-cdc-connector. You can try contributing your fix to the flink-cdc community.
感谢您提出此问题。这似乎是 flink-cdc-connector 的问题。你可以尝试将你的修复贡献给 flink-cdc 社区。 OK,I have submitted issus to them, and this issue can be closed FLINK-36164
Search before asking
Version
1.6.1
What's Wrong?
When synchronizing a PostgreSQL table using a connector, if the table contains a large number of partitions, the checkpoint always fails. By tracing the source code, it was found that the PostgresDialect's queryTableSchema queries the schema of each table. This schema querying during each checkpoint causes connection timeouts.
check point Exception:
error log:
What You Expected?
When synchronizing a PostgreSQL table using a connector, if the table contains a large number of partitions, the checkpoint always fails. By tracing the source code, it was found that the
PostgresDialect
'squeryTableSchema
queries the schema of each table. This schema querying during each checkpoint causes connection timeouts.Solution: Use caching. Each partitioned table should match and fetch the schema only once using the following parameters:
--multi-to-one-origin "jjdb_.*|fkdb_.*|pjdb_.*"
--multi-to-one-target "dwd_jjdb|dwdfkdb|dwd_pjdb"
How to Reproduce?
Steps to Reproduce:
Anything Else?
Method Modification: I modified the method as follows and it works well:
Are you willing to submit PR?
Code of Conduct