search

apache / celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.
https://celeborn.apache.org/
Apache License 2.0
893 stars 361 forks source link

[CELEBORN-1719] Introduce celeborn.client.spark.stageRerun.enabled with alternative celeborn.client.spark.fetch.throwsFetchFailure to enable spark stage rerun #2920

Closed SteNicholas closed 4 days ago

SteNicholas commented 1 week ago

What changes were proposed in this pull request?

  1. Introduce celeborn.client.spark.stageRerun.enabled with alternative celeborn.client.spark.fetch.throwsFetchFailure to enable spark stage rerun.
  2. Change the default value of celeborn.client.spark.fetch.throwsFetchFailure from false to true, which enables spark stage rerun at default.

Why are the changes needed?

User could not directly understand the meaning of celeborn.client.spark.fetch.throwsFetchFailure as whether to enable stage rerun, which means that client throws FetchFailedException instead of CelebornIOException. It's recommended to introduce celeborn.client.spark.stageRerun.enabled with alternative celeborn.client.spark.fetch.throwsFetchFailure to enable spark stage rerun.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI.

ErikFang commented 4 days ago

LGTM

SteNicholas commented 4 days ago

Merged to main(v0.6.0).