Why is Fulcrum using so much memory?

[hMsats]

Still a HUGE Fulcrum fan!

I run Fulcrum with Bitcoin Core on a server with 8GB of RAM. I gave 2500MB to bitcoind, so that should be about 31% of the total memory which corresponds well with the 27.3% shown below:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
user     1108261 20.4 27.3 7648024 2192612 ?     SLl  jul14 5403:45 ./bitcoind -datadir=/media/ssd/.bitcoin/
user     2869519 12.9 41.0 5147744 3287428 ?     Sl   aug01  86:55 /home/user/Fulcrum/Fulcrum /home/user/Fulcrum/config.conf

I gave 1GB of RAM to Fulcrum. I know that Fulcrum can take a bit more memory but the 41% shown above corresponds to 3.3GB of RAM. It's not a problem for my server but I don't understand it.

Do you have some explanation? Thanks in advance.

Extra info:

Ubuntu 20.04.6 LTS

Restarted Fulcrum yesterday, so it hasn't been running for a very long time.

Latest release:Fulcrum 1.9.1 (Release d628948)

db_max_open_files = 20 db_mem = 1000.0 (memory usage was about the same without db_mem set in config.conf)

[2023-08-01 19:54:59.806] Loaded SSL certificate: electrum.bitcoinserver.nl  expires: di september 5 2023 12:03:57
[2023-08-01 19:54:59.806] Loaded key type: private algorithm: RSA
[2023-08-01 19:54:59.806] Enabled JSON parser: simdjson
[2023-08-01 19:54:59.806] simdjson implementations:
[2023-08-01 19:54:59.806]     haswell: Intel/AMD AVX2  [supported]
[2023-08-01 19:54:59.806]     westmere: Intel/AMD SSE4.2  [supported]
[2023-08-01 19:54:59.806]     fallback: Generic fallback implementation  [supported]
[2023-08-01 19:54:59.806] active implementation: haswell
[2023-08-01 19:54:59.806] jemalloc: unavailable
[2023-08-01 19:54:59.806] Qt: version 5.12.8
[2023-08-01 19:54:59.806] rocksdb: version 6.14.6-ed43161
[2023-08-01 19:54:59.806] simdjson: version 0.6.0
[2023-08-01 19:54:59.806] ssl: OpenSSL 1.1.1f  31 Mar 2020
[2023-08-01 19:54:59.806] zmq: libzmq version: 4.3.2, cppzmq version: 4.7.1
[2023-08-01 19:54:59.806] Fulcrum 1.9.1 (Release d628948) - di aug. 1, 2023 19:54:59.806 CEST - starting up ...
[2023-08-01 19:54:59.806] Max open files: 1048576 (increased from default: 1024)
[2023-08-01 19:54:59.807] Loading database ...
[2023-08-01 19:55:00.103] DB memory: 1000.00 MiB
[2023-08-01 19:55:00.103] Coin: BTC
[2023-08-01 19:55:00.103] Chain: main
[2023-08-01 19:55:00.103] Verifying headers ...
[2023-08-01 19:55:02.651] Initializing header merkle cache ...
[2023-08-01 19:55:03.847] Checking tx counts ...
[2023-08-01 19:55:09.914] 873043395 total transactions
[2023-08-01 19:55:09.915] UTXO set: 114037678 utxos, 9579.165 MB
[2023-08-01 19:55:10.192] DB version: v2
[2023-08-01 19:55:10.192] BitcoinDMgr: starting 3 bitcoin RPC clients ...
[2023-08-01 19:55:10.192] BitcoinDMgr: started ok
[2023-08-01 19:55:10.192] Stats HTTP: starting 1 server ...
[2023-08-01 19:55:10.193] Starting listener service for HttpSrv 127.0.0.1:8080 ...
[2023-08-01 19:55:10.193] Service started, listening for connections on 127.0.0.1:8080
[2023-08-01 19:55:10.289] <Controller> Block height 801238, downloading new blocks ...
[2023-08-01 19:55:12.667] <Controller> Processed 1 new block with 3097 txs (7582 inputs, 9995 outputs, 14143 addresses), verified ok.
[2023-08-01 19:55:12.668] <Controller> Block height 801238, up-to-date
[2023-08-01 19:55:12.669] SrvMgr: starting PeerMgr ...

[cculianu]

Memory fragmentation. glibc's malloc allocator fragments memory and fulcrum does many allocations and free's in many threads. See if you can link it to use jemalloc. Maybe install jemalloc and recompile. It will help.

[cculianu]

If, after you compile and link to jemalloc, and Fulcrum shows it found jemalloc at runtime, and you are still unsatisfied with the memory consumption, then you can tell jemalloc to behave itself and not waste memory by having the following environment variable set:

MALLOC_CONF=tcache:false

Note that the above comes with a small performance penalty but it forces jemalloc to clean up after itself in per-thread allocations.

[hMsats]

Thanks for the fast reply! I will try that but not today because today I'm very busy. Will report back soon.

[hMsats]

Found some time today. Installed jemalloc and compiled Fulcrum with make clean; qmake LIBS+=-ljemalloc; make -j4 and it seems to work! Fulcum outputs it was using [2023-08-02 10:51:19.164] jemalloc: version 5.3.0-113-g62648c8.

19.9% is much better than 41%:

user     1108261 20.3 27.6 7405000 2215504 ?     SLl  jul14 5464:15 ./bitcoind -datadir=/media/ssd/.bitcoin/
user     3076974 11.1 19.9 3511792 1600088 ?     Tl   10:51  15:14 /home/user/Fulcrum/Fulcrum /home/user/Fulcrum/config.conf

and it doesn't seem to increase. Thanks a lot. Genius!

BTW in Fulcrum.pro jemalloc is written as jemaloc in a comment:

• User can override auto-detection by specifying "LIBS+=-ljemaloc..." on the => - User can override auto-detection by specifying "LIBS+=-ljemalloc..." on the

I'll keep an eye on the memory usage but for now closing this issue.

[cculianu]

Glad it all worked out. Yeah in my experience consumption remains stable.

You may find it will be even lower with that env var I talked about above but the jemalloc docs say then allocations will use a process-wide lock which may mean parallelism is sacrificed to some degree.

[hMsats]

Yeah in my experience consumption remains stable.

After running Fulcrum for more than 5 days, I can confirm that memory usage is stable and much lower now.

[cculianu]

Yay! And the Fulcrum admins of the world rejoice!