search

apache / bookkeeper

Apache BookKeeper - a scalable, fault tolerant and low latency storage service optimized for append-only workloads
https://bookkeeper.apache.org/
Apache License 2.0
1.9k stars 902 forks source link

Bookie GC pressure when use ReadEntryProcessorV3 #3428

Open horizonzy opened 2 years ago

horizonzy commented 2 years ago

BUG REPORT When we get entry from bookie in ReadEntryProcessorV3#readEntry, the response is DirectByteBuf, the v3 use proto buf response. So it will copy byte array to proto buf builder, it will use heap memory.

If the user read request is frequent, it will introduce gc pressure.

https://github.com/apache/bookkeeper/blob/ca7d7c23e3aa572befeb447157b3ef9d96bb1449/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/ReadEntryProcessorV3.java#L179

    public static ByteString copyFrom(ByteBuffer bytes, int size) {
        checkRange(0, size, bytes.remaining());
        byte[] copy = new byte[size];
        bytes.get(copy);
        return new LiteralByteString(copy);
    }
dlg99 commented 2 years ago

there are a lot of places like that for v3 requests (grep for ByteString.copyFrom). As I understand. that at leas a part of the reason for Pulsar to use v2 protocol https://github.com/apache/pulsar/blob/89b6a535aa8de8e2475cd39320897700e64d269e/conf/broker.conf#L870

You can try using UnsafeByteOperations.unsafeWrap instead of ByteString.copyFrom (e.g. see https://github.com/apache/bookkeeper/pull/1361 ) to reduce memory pressure but you have to be really careful about not recycling entryBody (and similar) before the protobuf-related operation completes.