Abnormal memory growth

[enefuture]

Note: for reported bugs, please fill in the following details. bug reports without detailed steps on how to reproduce will be automatically closed.

Dragonboat version

v3.3.4

Expected behavior

Actual behavior

Abnormal memory growth

Steps to reproduce the behavior

benchmark cluster with 1kb value

[lni]

@enefuture

any chance you can provide more details on what you did for such benchmark? providing a go memory profile can also greatly help. the profiler should be able to tell what is actually on the go heap using those 50GB memory.

[lni]

I checked memory usage using the tool/checkdisk tool. By making huge number of 1kbytes proposals, memory usage is always lower than 10GBytes with the majority of on heap usage being spent on various internal buffers that can be reduced by user configurations.

See my memory profile below.

go tool pprof mem.pprof
File: checkdisk
Build ID: 54b15ce5af0a11c48c15383648b31b5746084a51
Type: **inuse_space**
Time: Feb 14, 2022 at 12:23pm (UTC)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top30
Showing nodes accounting for 2.24GB, 96.58% of 2.32GB total
Dropped 163 nodes (cum <= 0.01GB)
Showing top 30 nodes out of 62
      flat  flat%   sum%        cum   cum%
       2GB 86.06% 86.06%        2GB 86.06%  github.com/lni/dragonboat/v3/internal/logdb.newContext (inline)
    0.06GB  2.51% 88.57%     0.06GB  2.51%  github.com/lni/dragonboat/v3/internal/server.NewMessageQueue (inline)
    0.04GB  1.91% 90.48%     0.04GB  1.91%  github.com/lni/dragonboat/v3/raftpb.(*Entry).unmarshal
    0.03GB  1.41% 91.89%     0.03GB  1.41%  github.com/cockroachdb/pebble.(*Batch).grow
    0.03GB  1.35% 93.23%     0.03GB  1.35%  github.com/lni/dragonboat/v3.newEntryQueue (inline)
    0.03GB  1.32% 94.56%     0.03GB  1.32%  runtime.allocm
    0.02GB  0.72% 95.28%     0.06GB  2.72%  github.com/lni/dragonboat/v3/internal/transport.(*TCP).serveConn
    0.02GB   0.7% 95.98%     0.02GB   0.7%  github.com/cockroachdb/pebble/internal/record.(*LogWriter).queueBlock
    0.01GB   0.6% 96.58%     0.01GB   0.6%  github.com/lni/dragonboat/v3/internal/transport.NewTCPConnection
         0     0% 96.58%     0.03GB  1.41%  github.com/cockroachdb/pebble.(*Batch).Set
         0     0% 96.58%     0.03GB  1.41%  github.com/cockroachdb/pebble.(*Batch).SetDeferred (inline)
         0     0% 96.58%     0.03GB  1.41%  github.com/cockroachdb/pebble.(*Batch).prepareDeferredKeyValueRecord
         0     0% 96.58%     0.02GB  0.76%  github.com/cockroachdb/pebble.(*DB).Apply
         0     0% 96.58%     0.02GB  0.76%  github.com/cockroachdb/pebble.(*DB).commitWrite
         0     0% 96.58%     0.02GB  0.76%  github.com/cockroachdb/pebble.(*commitPipeline).Commit
         0     0% 96.58%     0.02GB  0.76%  github.com/cockroachdb/pebble.(*commitPipeline).prepare
         0     0% 96.58%     0.02GB   0.7%  github.com/cockroachdb/pebble/internal/record.(*LogWriter).SyncRecord
         0     0% 96.58%     0.02GB   0.7%  github.com/cockroachdb/pebble/internal/record.(*LogWriter).emitFragment
         0     0% 96.58%     0.01GB  0.57%  github.com/lni/dragonboat/v3.(*NodeHost).Propose
         0     0% 96.58%     0.10GB  4.37%  github.com/lni/dragonboat/v3.(*NodeHost).StartCluster
         0     0% 96.58%     2.01GB 86.38%  github.com/lni/dragonboat/v3.(*NodeHost).createLogDB
         0     0% 96.58%     0.01GB  0.57%  github.com/lni/dragonboat/v3.(*NodeHost).propose
         0     0% 96.58%     0.10GB  4.37%  github.com/lni/dragonboat/v3.(*NodeHost).startCluster
         0     0% 96.58%     0.07GB  2.87%  github.com/lni/dragonboat/v3.(*engine).processSteps
         0     0% 96.58%     0.07GB  2.87%  github.com/lni/dragonboat/v3.(*engine).stepWorkerMain
         0     0% 96.58%     0.01GB  0.55%  github.com/lni/dragonboat/v3.(*node).propose
         0     0% 96.58%     0.01GB  0.55%  github.com/lni/dragonboat/v3.(*pendingProposal).propose
         0     0% 96.58%     0.01GB  0.55%  github.com/lni/dragonboat/v3.(*proposalShard).propose
         0     0% 96.58%     2.01GB 86.40%  github.com/lni/dragonboat/v3.NewNodeHost
         0     0% 96.58%     0.07GB  2.87%  github.com/lni/dragonboat/v3.newExecEngine.func1

Please note that your "benchmark" program including its 3rd party dependencies might leak memory.

Will close this issue in a week unless a memory profile can be provided to show that this is caused by dragonboat. I'd also ask all other users to provide full details when reporting issues - no one is responsible for debugging your program for free.

[lni]

Closed due to the lack of essential info.

Please feel free to reopen if above requested details can be provided