Long-running node excessive memory usage

[LesnyRumcajs]

Issue summary

An issue with a long-running calibnet node surfaced in which Forest is using significantly more memory than it should. This might be related to https://github.com/ChainSafe/forest/issues/4346

We can see that the memory grows up until 13 GiB, then the node likely gets restarted (can you please confirm @samuelarogbonlo?). Now, this is nowhere close our expectations where the calibnet node should run easily on a RPi4.

Is there a memory leak?

logs.txt

[client]
#Controls whether the keystore is encrypted
encrypt_keystore = false
data_dir = "/home/forest/forest_data"
enable_rpc = true
enable_metrics_endpoint = true
metrics_address = "127.0.0.1:6116"
rpc_address = "0.0.0.0:2345"

[network]
listening_multiaddrs = ["/ip4/0.0.0.0/tcp/34000"]
#Kademlia option enforce p2p connections.
kademlia = true
#Amount of peers the node should maintain a connection with
#Helps in syncing ( moderate value to not be exceeded)
target_peer_count = 5000

Forest v0.19.0, calibration network bootstrap node

Other information and links

[ruseinov]

Yeah, this is really weird. I have about the same memory consumption for a mainnet node, on a machine limited to 16GB RAM.