Currently Paimon writer operators in Flink have states, which record the commitUser (for all writers) and the list of active buckets (for lookup or full-compaction changelog producer). These states prevent a writer object to be closed too early and cause conflicts.
However for append only unaware bucket writers, because they only create new files, there is no conflict. So for these writers we can remove Flink state. This optimization is also useful for bounded stream jobs, because when chaining source and write operator together, if some source parallelism have finished, checkpoint cannot be performed if there are union list states in the operators which are still running.
wwj6591812
commented
2 months ago
Purpose
Currently Paimon writer operators in Flink have states, which record the
commitUser(for all writers) and the list of active buckets (for lookup or full-compaction changelog producer). These states prevent a writer object to be closed too early and cause conflicts.However for append only unaware bucket writers, because they only create new files, there is no conflict. So for these writers we can remove Flink state. This optimization is also useful for bounded stream jobs, because when chaining source and write operator together, if some source parallelism have finished, checkpoint cannot be performed if there are union list states in the operators which are still running.
Tests
IT case.
API and Format
No format changes.
Documentation
No new feature.