[Flink] Allow disable auto schema change. Add batch size config

lakesoul-io / LakeSoul

LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.

https://lakesoul-io.github.io/

Apache License 2.0

2.38k stars 424 forks source link

[Flink] Allow disable auto schema change. Add batch size config #528

Closed xuchen-plus closed 2 months ago

xuchen-plus commented 2 months ago

Added a new config AUTO_SCHEMA_CHANGE(lakesoul.sink.auto_schema_change), default false. It is only enabled in buildLakeSoulDMLSink and buildArrowSink for sink to multiple tables.
Set batch size for reader from config BATCH_SIZE