Closed jeqo closed 1 month ago
However, in Java, the close method of ByteArrayInputStream has no effect. The methods of this class can be called after the stream has been closed without generating an IOException. This is because the data of ByteArrayInputStream is stored in memory, unlike file streams or network streams that require actual resource cleanup, so there may not be an out-of-memory problem
@funky-eyes good catch! I'm adding more changes on how bytes are to the request using byte buffers only instead of array copying. PTAL
For reference, similar improvements have been implemented on kafka core: https://github.com/apache/kafka/pull/15589
Use only ByteRange slices to pass bytes to S3 client operations and remove array copying.
Doing some benchmarks on the current implementation, the Arrays.copyOfRange dominates the memory allocation:
By switching to the ByteBuffer approach, this copying is removed: