apache / incubator-uniffle

Uniffle is a high performance, general purpose Remote Shuffle Service.
https://uniffle.apache.org/
Apache License 2.0
387 stars 149 forks source link

[Improvement] [remote merge] remove cached block for partition. #2171

Closed zhengchenyu closed 1 month ago

zhengchenyu commented 1 month ago

Code of Conduct

Search before asking

What would you like to be improved?

I introduce cachedblockMap and mergedBlockMap in org.apache.uniffle.server.merge.Partition. It is used to get blocks quickly by map like data structure. Why I introduce this? Because blocks in ShuffleBuffer is linked list before. But after #1763, blocks in ShuffleBufferWithSkipList have been map like data structure. So remove cached block, and save memory. So after this, only ShuffleBufferWithSkipList is supported in remote merge mode.

How should we improve?

No response

Are you willing to submit PR?