search

apache / celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.
https://celeborn.apache.org/
Apache License 2.0
863 stars 351 forks source link

[CELEBORN-1519] Do not update estimated partition size if it is unchanged #2642

Closed s0nskar closed 1 month ago

s0nskar commented 1 month ago

What changes were proposed in this pull request?

We will not update the estimated partition size if it is unchanged.

Why are the changes needed?

Celeborn currently triggers an workerinfo update even-though the estimated partition size is not changed. This leads to unnecessary logging and redundant worker info update operations.

Example log -

[master-partition-size-updater] WARN org.apache.celeborn.service.deploy.master.clustermeta.AbstractMetaManager - Celeborn cluster estimated partition size changed from 64.0 MiB to 64.0 MiB

Does this PR introduce any user-facing change?

NO

How was this patch tested?

Existing UT's

pan3793 commented 1 month ago

Thanks, merged to main