Flink: make `StatisticsOrRecord` to be correctly serialized and deser…

apache / iceberg

Apache Iceberg

https://iceberg.apache.org/

Apache License 2.0

6.49k stars 2.24k forks source link

Flink: make `StatisticsOrRecord` to be correctly serialized and deser… #11557

Open huyuanfeng2018 opened 1 week ago

huyuanfeng2018 commented 1 week ago

background

When I configure distribution-mode=RANGE in the flink task, the task can process the data as expected, but the processing speed is insufficient. Like #7393, there is a serialization problem.

Change part

Add StatisticsOrRecordTypeInformation and specify it as outTypeInfo
add test org.apache.iceberg.flink.sink.shuffle.TestStatisticsOrRecordTypeInformation

huyuanfeng2018 commented 1 week ago

@stevenzwu @pvary can you review it for me, thank you very much !