search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.19k stars 3.82k forks source link

kvserver: optimize raft entry encoding #134936

Open tbg opened 2 weeks ago

tbg commented 2 weeks ago

We currently store raft entries in nested protobufs^1 (RaftMessageRequest contains raftpb.Message contains []Entry contains in its Data []byte field (after stripping a prefix) a kvserverpb.RaftCommand^2. This is all wildly inefficient, I think when we looked at it we found a ~12x allocation overhead (i.e. for one useful byte we allocated 12x as much).

There are various ways to improve this, but one could be adding a new entry encoding in which we use a flatbuffer instead of kvserverpb.RaftCommand first (which removes the nested unmarshal), and later possibly replace RaftMessageRequest with a flatbuffer as well. However, unless we also use a flatbuffer codec for gRPC, we'll always do some inefficient unmarshaling.

Epic: CRDB-42584

Jira issue: CRDB-44302

tbg commented 2 weeks ago

Here's a picture from the vaults[^1], with some indication of which allocations could be avoided (fbs means flatbuffers here)

[^1]: jamboard link on this thread

Untitled Jam 1

tbg commented 3 days ago

Here's a more recent picture from master running oltp_write_only.

image

allocs30s.pb.gz https://pprof.me/b820d13219711c30401cb7803f01bb01