search

apache / iceberg

Apache Iceberg
https://iceberg.apache.org/
Apache License 2.0
6.43k stars 2.22k forks source link

Improve retry algorithm for Tasks #7528

Open jackye1995 opened 1 year ago

jackye1995 commented 1 year ago

Feature Request / Improvement

Currently we just use exponential backoff. In practice when there are many concurrent writes, the conflict rate is still high even with jitter. There are some other algorithms like AIMD that was proven to be effective, and might be worth considering to add to Tasks to retry commits.

Reference: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-emrfs-retry.html

Query engine

None

singhpk234 commented 1 year ago

+1 for this, it will be awesome if we could make the retry pluggable as well :) !

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

github-actions[bot] commented 11 months ago

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'