StarRocks / starrocks-connector-for-apache-flink

Apache License 2.0
195 stars 160 forks source link

[Feature] Sink supports LZ4 compression with json format #354

Closed banmoy closed 7 months ago

banmoy commented 7 months ago

What type of PR is this:

Which issues of this PR fixes :

After StarRocks supports lz4 compression for stream load json format in https://github.com/StarRocks/starrocks/pull/43732, the connector can compress the json data before sending to StarRocks which will reduce the network traffic significantly. In the test to load clickbench data to starrocks, the compression ratio can be ~8, and the load performance has a 3.64% degradation which is acceptable. You can enable it with the following configuration

'sink.properties.format' = 'json'
'sink.properties.compression' = 'lz4_frame'

Problem Summary(Required) :

Checklist: