[CELEBORN-1561] Make compressor closable and close native Lz4Compressor when ShuffleClient shutdown - Githubissues

apache / celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.

https://celeborn.apache.org/

Apache License 2.0

862 stars 351 forks source link

[CELEBORN-1561] Make compressor closable and close native Lz4Compressor when ShuffleClient shutdown #2681

Closed bowenliang123 closed 1 month ago

bowenliang123 commented 1 month ago

What changes were proposed in this pull request?

close the Compressor instance when ShuffleClient shutdown
as the Lz4Compressor may create native compressor instance when liblz4-java.so available on OS, the StreamingXXHash32JNI class provides the close method for memory releasing.

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?