PowerDNS / lightningstream

Lightning Stream syncs LMDB databases through S3 buckets between multiple servers, including PowerDNS Authoritative server 4.8+ LMDBs
https://doc.powerdns.com/lightningstream/
MIT License
25 stars 15 forks source link

Reduce memory used for when loading a snapshot #30

Closed wojas closed 1 year ago

wojas commented 1 year ago

The generated protobuf code made copies of all key/value byte slices. This patches the generated code to directly return the byte slices.

In local testing, this reduced memory allocations when loading snapshots by half. PR #31 will provide bigger savings later.

Next step is to rewrite the protobuf handling to iterate over the protobuf data instead of creating the []KV slices, but that requires a significant rewrite.

Additionally, this enables the net/http/pprof endpoints for debugging.

Output of go tool pprof http://127.0.0.1:9152/debug/pprof/heap and then top 10 below, with a test set of 1M domains and 6M records.

Before:

      flat  flat%   sum%        cum   cum%
 2047.55MB 56.26% 56.26%  2722.09MB 74.80%  powerdns.com/platform/lightningstream/snapshot.(*DBI).Unmarshal
  674.54MB 18.54% 98.45%   674.54MB 18.54%  powerdns.com/platform/lightningstream/snapshot.(*KV).Unmarshal

After:

 1378.77MB 52.20% 52.20%  1378.77MB 52.20%  powerdns.com/platform/lightningstream/snapshot.(*DBI).Unmarshal
nvaatstra commented 1 year ago

LGTM; Nice improvement