After StarRocks supports lz4 compression for stream load json format in https://github.com/StarRocks/starrocks/pull/43732, the connector can compress the json data before sending to StarRocks which will reduce the network traffic significantly. In the test to load clickbench data to starrocks, the compression ratio can be ~8, and the load performance has a 3.64% degradation which is acceptable.
You can enable it with the following configuration
What type of PR is this:
Which issues of this PR fixes :
After StarRocks supports lz4 compression for stream load json format in https://github.com/StarRocks/starrocks/pull/43732, the connector can compress the json data before sending to StarRocks which will reduce the network traffic significantly. In the test to load clickbench data to starrocks, the compression ratio can be ~8, and the load performance has a 3.64% degradation which is acceptable. You can enable it with the following configuration
Problem Summary(Required) :
Checklist: