apache / amoro

Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats.
https://amoro.apache.org/
Apache License 2.0
863 stars 288 forks source link

[Feature]: Support insert part of columns of mixed-format table with flink in streaming mode #256

Open zhoujinsong opened 2 years ago

zhoujinsong commented 2 years ago

Description

Support insert part of columns of arctic table with flink in streaming mode

Use case/motivation

Arctic only support insert all table columns with flink in streaming mode now. If souce table DOES NOT have all columns which sink table need, user must join multiple source table, which will cause a big state problem. If Arctic support insert part of columns, user can start multiple flink job and each job insert part of columns(including join column) into arctic table to avoid problem mentioned above.

Describe the solution

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

shidayang commented 2 years ago

Design doc:https://docs.google.com/document/d/1CwvXGt--MZG9XcqN-Q4scqoLerGcY6seIgAOhPVv2BM/edit?usp=sharing

shidayang commented 2 years ago

Discussed in https://github.com/NetEase/arctic/discussions/494

zhoujinsong commented 11 months ago

Closed due to a change in plans.