ParquetWriter leaks memory

[cipriangerea]

The constructor for ParquetWriter is allocating a compressor at https://github.com/linkedin/iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java#L92

The compressor needs to be released once the writer is closed. If this doesn't happen we are leaking the native memory associated with the compressor.

[renshangtao]

Hi , i got the same question with you. I use flink session mode to run some job，and after several jobs finished , i used top command to see the memory of linux，i foud the memory is only a little left，and the taskmanager will be kill by the Linux because of oom

But i get what you have modify, and test again, the question is also here.

I see the code HeapBytesCompressor(CompressionCodecName codecName) { this.codecName = codecName; this.codec = getCodec(codecName); if (codec != null) { this.compressor = CodecPool.getCompressor(codec); this.compressedOutBuffer = new ByteArrayOutputStream(pageSize); } else { this.compressor = null; this.compressedOutBuffer = null; } }

public ByteArrayOutputStream(int size) { if (size < 0) { throw new IllegalArgumentException("Negative initial size: "

• size); } buf = new byte[size]; //here it only new size byte of memroy needn't free }

[Kontinuation]

It was fixed by https://github.com/apache/iceberg/pull/5126 I believe, maybe we can close this issue.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions[bot]]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'