search

itinycheng / flink-connector-clickhouse

Flink SQL connector for ClickHouse. Support ClickHouseCatalog and read/write primary data, maps, arrays to clickhouse.
Apache License 2.0
373 stars 156 forks source link

查询where条件没有下推到数据库执行 #76

Closed lapDog closed 1 year ago

lapDog commented 1 year ago

1.数据库表设计分区字段为 PARTITION BY toYYYYMMDD(device_time)； 2.scan.partition.column这个字段我不知咋配了。。。是直接写device_time? 3.flink sql注册表后执行简单的select*from xx where device_time ...直接卡住半天，没有到数据库中执行。。应该走的是纯内存了。。 大佬帮忙看下呢

itinycheng commented 1 year ago

1.数据库表设计分区字段为 PARTITION BY toYYYYMMDD(device_time)； 2.scan.partition.column这个字段我不知咋配了。。。是直接写device_time? 3.flink sql注册表后执行简单的select*from xx where device_time ...直接卡住半天，没有到数据库中执行。。应该走的是纯内存了。。 大佬帮忙看下呢

  1. 直接使用列名：scan.partition.column = device_time，这个是配合scan.partition.lower-bound, scan.partition.upper-bound等做数据分区并行读取的，只支持number类型的列；
  2. select*from xx where device_time ...这里边的filter没下推和上边的是两个问题，filter没下推有两种可能，flink optimizer问题/connector不支持Filter；本地做个breakpoint test看看具体什么原因；

FilterPushDown refer to: https://github.com/itinycheng/flink-connector-clickhouse/blob/master/src/main/java/org/apache/flink/connector/clickhouse/util/FilterPushDownHelper.java