apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.35k stars 928 forks source link

[cdc] optimize database sync table filter logic #3654

Open MOBIN-F opened 3 months ago

MOBIN-F commented 3 months ago

Search before asking

Paimon version

master

Compute Engine

flink

Minimal reproduce step

none

What doesn't meet your expectations?

In the current database_sync logic, if the including_tables or excluding_tables parameters are set, a series of parsing and conversion logic will still be performed on the tables that need to be filtered. We should filter them as soon as we can get the tableName to avoid some meaningless parsing and conversion operations.

Anything else?

No response

Are you willing to submit a PR?