apache / incubator-uniffle

Uniffle is a high performance, general purpose Remote Shuffle Service.
https://uniffle.apache.org/
Apache License 2.0
387 stars 149 forks source link

[#2173] feat(remote merge): support netty for remote merge. #2202

Closed zhengchenyu closed 1 month ago

zhengchenyu commented 1 month ago

What changes were proposed in this pull request?

Support netty for remote merge. Use direct ByteBuf to replace with byte[] when netty is enable. And optimized code structure to avoid memory leaks

Why are the changes needed?

Fix: #2173

Does this PR introduce any user-facing change?

No.

How was this patch tested?

unit test, integration test, real job in cluster.

github-actions[bot] commented 1 month ago

Test Results

 2 926 files  ±  0   2 926 suites  ±0   6h 26m 6s :stopwatch: + 16m 25s  1 088 tests + 39   1 086 :white_check_mark: + 39   2 :zzz: ±0  0 :x: ±0  13 630 runs  +585  13 600 :white_check_mark: +585  30 :zzz: ±0  0 :x: ±0 

Results for commit 60659ce3. ± Comparison against base commit 87f9b6f7.

This pull request removes 20 and adds 59 tests. Note that renamed tests count towards both. ``` org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile1{String, File}[1] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile1{String, File}[2] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile2{String, File}[1] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile2{String, File}[2] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile3{String, File}[1] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile3{String, File}[2] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile4{String, File}[1] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile4{String, File}[2] org.apache.uniffle.common.serializer.PartialInputStreamTest ‑ testReadFileInputStream org.apache.uniffle.common.serializer.PartialInputStreamTest ‑ testReadMemroyInputStream … ``` ``` org.apache.uniffle.common.merger.MergerTest ‑ testMergeSegmentToFile{String, File}[2] org.apache.uniffle.common.merger.MergerTest ‑ testMergeSegmentToFile{String, File}[3] org.apache.uniffle.common.merger.MergerTest ‑ testMergeSegmentToFile{String, File}[4] org.apache.uniffle.common.netty.protocol.NettyProtocolTest ‑ testGetSortedShuffleDataRequest org.apache.uniffle.common.netty.protocol.NettyProtocolTest ‑ testGetSortedShuffleDataResponse org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFileUseDirect{String, File}[1] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFileUseDirect{String, File}[2] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile{String, File}[1] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile{String, File}[2] org.apache.uniffle.common.records.RecordsReaderWriterTest ‑ testWriteAndReadRecordFile{String, File}[3] … ```

:recycle: This comment has been updated with latest results.

zhengchenyu commented 1 month ago

@jerqi Can you please review this PR?