apache / incubator-uniffle

Uniffle is a high performance, general purpose Remote Shuffle Service.
https://uniffle.apache.org/
Apache License 2.0
387 stars 149 forks source link

[FEATURE] Bitmap in shuffle server contains multiple partitions' blocks causing performance issues. #2219

Closed lwllvyb closed 4 days ago

lwllvyb commented 1 month ago

Code of Conduct

Search before asking

Describe the feature

In our internal clusters, we found the shuffle server performance will decrease because the bitmap contains multiple partitions' blocks. Although we can use rss.client.bitmap.splitNum to split the bitmap, there will still be multiple partitions sharing the bitmap. Concurrent access the bitmap will cause performance degradation.

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?