The generated protobuf code made copies of all key/value byte slices.
This patches the generated code to directly return the byte slices.
In local testing, this reduced memory allocations when loading snapshots
by half. PR #31 will provide bigger savings later.
Next step is to rewrite the protobuf handling to iterate over the
protobuf data instead of creating the []KV slices, but that requires a
significant rewrite.
Additionally, this enables the net/http/pprof endpoints for debugging.
Output of go tool pprof http://127.0.0.1:9152/debug/pprof/heap and then top 10 below, with a test set of 1M domains and 6M records.
The generated protobuf code made copies of all key/value byte slices. This patches the generated code to directly return the byte slices.
In local testing, this reduced memory allocations when loading snapshots by half. PR #31 will provide bigger savings later.
Next step is to rewrite the protobuf handling to iterate over the protobuf data instead of creating the []KV slices, but that requires a significant rewrite.
Additionally, this enables the
net/http/pprof
endpoints for debugging.Output of
go tool pprof http://127.0.0.1:9152/debug/pprof/heap
and thentop 10
below, with a test set of 1M domains and 6M records.Before:
After: