Improve iceberg commit logic to so that we can avoid restarting streaming job when commit conflicts happens.

risingwavelabs / risingwave

SQL stream processing, analytics, and management. We decouple storage and compute to offer efficient joins, instant failover, dynamic scaling, speedy bootstrapping, and concurrent query serving.

https://www.risingwave.com/slack

Apache License 2.0

6.62k stars 543 forks source link

Improve iceberg commit logic to so that we can avoid restarting streaming job when commit conflicts happens. #13155

Open liurenjie1024 opened 8 months ago

liurenjie1024 commented 8 months ago

According to user's comment, paimon supports concurrent writes, but it doesn't work well when facing committing conflicts, which may lead to restart of flink streaming job. We should try to avoid job restarting when facing committing conflicts.

github-actions[bot] commented 1 week ago

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean. Don't worry if you think the issue is still valuable to continue in the future. It's searchable and can be reopened when it's time. 😄