OOM / High Memory Use

[AngusP]

Running Umbrel 0.5.3 on a 2019 Dell Optiplex 3060 (Intel i3-8100T CPU @ 3.10GHz) with an otherwise vanilla install of Debian 11. System memory 8GB, no swap (for drive longevity, and 8GB should be fine if it works on a 4GB Pi?). Debian install is pretty normal, except I went for LUKS + Btrfs rather than naked Ext4.

Linux umbrel 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux

I initially noticed something was up because the Pi-Hole slowed down a lot so DNS resolution was slow.

Checking htop in a shell, it looked like the culprit was mainly bitcoind running around 20 threads and a RSS of sorry I don't remember. The system was pegged at load average ~50 and memory usage ~99% (~7.95GB). Didn't look like things were getting OOM killed though, but perhaps they were.

Upon restarting Umbrel (stop then start scripts) it also looks like I lost around two days of blocks. My Bitcoin node was synced, but now is at "779,417 of 779,625 blocks" (ish, it's been catching up for about ten minutes already)

Post-restart memory and load is still pretty high but a bit better.

I have Tor, I2P, Inbound enabled (though I think inbound isn't working because I need to poce some UPnP settings elsewhere). Cache is default 300MB. Mainnet, full-RBF on.

Other apps: Bitfeed, Electrs, Jam, LND, Lightning Shell, Lightning Terminal, LNBits, Mempool, Pi-hole, RTL, Thunderhub, Torq.

Happy to provide more info as I guess this probably isn't enough to find a solution unless someone has an "ah-ha!" moment -- other than seeing if this re-occurs in a day or so and I can grab some more details.

Apologies for not screen-grabbing the before-restart state, but this is how it's running now having finished sync again:

My gut feeling is this is still pretty high for memory use?

[ademar111190]

I have Umbrel running on no pi 4 and I'm noticing every time I open the settings page the memory usage by Umbrel itself is bigger, after around 40 hours it stops working and I need to restart the pi. I think there is a memory leak somewhere on Umbrel. My pi is running the official raspberry pi image, the only software I installed was Umbrel and apps through the Umbrel store.

How it looks now as I just restarted. I'll add another screenshot later today or earlier tomorrow.

Edited: here how memory consumption has grown 11 hours later.

Edit 2nd time: more or less 24 hours after the restart, the memory usage still growing.

Edit 3rd time: more or less 34 hours after the restart, the memory usage still growing.

[jimbrend]

Hey there! Can you SSH and provide output of this command to see what is taking up the space?

ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | head -n 10

[ademar111190]

@JimSB The PI stop to respond again; it has been almost two days already, so no surprise. I'll restart, and tomorrow I'll run that command and share the output here.

[ademar111190]

I add here the output as time goes on.

First, few minutes after the restart:

  0.00 Mb COMMAND
858.79 Mb /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
521.53 Mb mysqld
402.84 Mb bitcoind -port=8333 -rpcport=8332 -rpcbind=10.21.21.8 -rpcbind=127.0.0.1 -rpcallowip=10.21.0.0/16 -rpcallowip=127.0.0.1 -rpcauth=umbrel:d34a498c1507c9f2a491fa494e9eed01$7ace22fe0590bc02ec7a31648cc9582c4b447c009b82dcbeadd79e258248d0dc -zmqpubrawblock=tcp://0.0.0.0:28332 -zmqpubrawtx=tcp://0.0.0.0:28333 -zmqpubhashblock=tcp://0.0.0.0:28334 -zmqpubsequence=tcp://0.0.0.0:28335 -blockfilterindex=1 -peerbloomfilters=1 -peerblockfilters=1 -rpcworkqueue=128
355.23 Mb node dist/main.js
168.45 Mb node dist/apps/immich/apps/immich/src/main
158.86 Mb /usr/bin/containerd
154.30 Mb node dist/apps/microservices/apps/microservices/src/main
134.41 Mb node /backend/dist/index.js
129.99 Mb /usr/local/bin/node ./bin/www

Second, more or less 24 hours after the restart:

  0.00 Mb COMMAND
943.81 Mb bitcoind -port=8333 -rpcport=8332 -rpcbind=10.21.21.8 -rpcbind=127.0.0.1 -rpcallowip=10.21.0.0/16 -rpcallowip=127.0.0.1 -rpcauth=umbrel:d34a498c1507c9f2a491fa494e9eed01$7ace22fe0590bc02ec7a31648cc9582c4b447c009b82dcbeadd79e258248d0dc -zmqpubrawblock=tcp://0.0.0.0:28332 -zmqpubrawtx=tcp://0.0.0.0:28333 -zmqpubhashblock=tcp://0.0.0.0:28334 -zmqpubsequence=tcp://0.0.0.0:28335 -blockfilterindex=1 -peerbloomfilters=1 -peerblockfilters=1 -rpcworkqueue=128
860.54 Mb /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
522.87 Mb mysqld
404.68 Mb /usr/sbin/dhcpcd -b -q
355.23 Mb node dist/main.js
159.36 Mb /usr/bin/containerd
147.81 Mb node /backend/dist/index.js
125.25 Mb /usr/local/bin/node ./bin/www
114.20 Mb node /opt/yarn-v1.22.19/bin/yarn.js start

One last time today (±32hours) as I think tomorrow I'll need to restart:

  0.00 Mb COMMAND
931.60 Mb bitcoind -port=8333 -rpcport=8332 -rpcbind=10.21.21.8 -rpcbind=127.0.0.1 -rpcallowip=10.21.0.0/16 -rpcallowip=127.0.0.1 -rpcauth=umbrel:d34a498c1507c9f2a491fa494e9eed01$7ace22fe0590bc02ec7a31648cc9582c4b447c009b82dcbeadd79e258248d0dc -zmqpubrawblock=tcp://0.0.0.0:28332 -zmqpubrawtx=tcp://0.0.0.0:28333 -zmqpubhashblock=tcp://0.0.0.0:28334 -zmqpubsequence=tcp://0.0.0.0:28335 -blockfilterindex=1 -peerbloomfilters=1 -peerblockfilters=1 -rpcworkqueue=128
860.79 Mb /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
522.87 Mb mysqld
511.89 Mb /usr/sbin/dhcpcd -b -q
355.23 Mb node dist/main.js
170.46 Mb node dist/apps/immich/apps/immich/src/main
170.01 Mb node dist/apps/microservices/apps/microservices/src/main
159.61 Mb /usr/bin/containerd
146.72 Mb node /backend/dist/index.js

The PI is still alive, more or less 44 hours later, so I managed to run the command one more time.

  0.00 Mb COMMAND
933.82 Mb bitcoind -port=8333 -rpcport=8332 -rpcbind=10.21.21.8 -rpcbind=127.0.0.1 -rpcallowip=10.21.0.0/16 -rpcallowip=127.0.0.1 -rpcauth=umbrel:d34a498c1507c9f2a491fa494e9eed01$7ace22fe0590bc02ec7a31648cc9582c4b447c009b82dcbeadd79e258248d0dc -zmqpubrawblock=tcp://0.0.0.0:28332 -zmqpubrawtx=tcp://0.0.0.0:28333 -zmqpubhashblock=tcp://0.0.0.0:28334 -zmqpubsequence=tcp://0.0.0.0:28335 -blockfilterindex=1 -peerbloomfilters=1 -peerblockfilters=1 -rpcworkqueue=128
861.29 Mb /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
591.32 Mb /usr/sbin/dhcpcd -b -q
522.87 Mb mysqld
355.23 Mb node dist/main.js
159.61 Mb /usr/bin/containerd
155.79 Mb node dist/apps/microservices/apps/microservices/src/main
149.73 Mb node /backend/dist/index.js
126.32 Mb /usr/local/bin/node ./bin/www

[AngusP]

For me: (uptime 8 days)

         0.00 Mb COMMAND 
       919.41 Mb node /backend/dist/index.js 
       854.99 Mb bitcoind -port=8333 -rpcport=8332 -rpcbind=10.21.21.8 -rpcbind=127.0.0.1 -rpcallowip=10.21.0.0/16 -rpcallowip=127.0.0.1 -rpcauth=umbrel:ffb5294424ddcec6dfaaa63613cf8773$266052ce0bcc298a99737eee6d2147f86f0778d4ab2df885fda59702ed650bb9 -zmqpubrawblock=tcp://0.0.0.0:28332 -zmqpubrawtx=tcp://0.0.0.0:28333 -zmqpubhashblock=tcp://0.0.0.0:28334 -zmqpubsequence=tcp://0.0.0.0:28335 -blockfilterindex=1 -peerbloomfilters=1 -peerblockfilters=1 -rpcworkqueue=128 
       804.33 Mb electrs 
       600.25 Mb lnd --configfile=/data/.lnd/umbrel-lnd.conf --listen=0.0.0.0:9735 --rpclisten=0.0.0.0:10009 --restlisten=0.0.0.0:8080 --bitcoin.active --bitcoin.mainnet --bitcoin.node=bitcoind --bitcoind.rpchost=10.21.21.8:8332 --bitcoind.rpcuser=umbrel --bitcoind.rpcpass=Zvb0JBTFE6yXeJvTJzVmu5tqWk8Lla2Zu1gywNCjLKU= --bitcoind.zmqpubrawblock=tcp://10.21.21.8:28332 --bitcoind.zmqpubrawtx=tcp://10.21.21.8:28333 --tor.active --tor.v3 --tor.control=10.21.21.11:29051 --tor.socks=10.21.21.11:9050 --tor.targetipaddress=10.21.21.9 --tor.password=MuIOJnqMgyRDLeBhUI-y3Z8f5pBQ9ErG8ODfwAZyVwM= 
       530.85 Mb mysqld 
       444.20 Mb /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock 
       166.96 Mb /usr/bin/gnome-shell 
       165.36 Mb node ./bin/www 
       154.89 Mb /usr/bin/containerd

[tlindi]

@ademar111190 "official raspberry pi image, the only software I installed was Umbrel" I'm running Umbrer from their official image just fine. Might be able to ditch RaspiOS and try Umbrel image? (Running Umbrel on 4B 4GB and 8GB without of issues. Uptimes ~50days+)

[technotion]

I have been experiencing similar behavior. Running Umbrel on a Linux VM/Ubuntu 22. Originally gave it 8GB of RAM and Umbrel used 97% of the RAM. I shutdown the VM, gave it 16GB of RAM. On restart, I watched the RAM usage get eaten up over the course of a few minutes. The VM is now using 97% of the 16GB. Something is not right. Running Umbrel, Bitcoin Node, Electrum, Mempool, and Bitfeed. No other services running on Umbrel.

[davotoula]

I had a similar issue last year. I narrowed it down to dhcpcd.

I configured my system to use static ip rather than dhcp. The problem went away.

[jimbrend]

Thanks for sharing @davotoula there's a guide for setting up static IP for anyone still having an issue you can see if this also helps: https://community.getumbrel.com/t/how-to-set-a-static-ip-for-your-umbrel-server/845 Let me know if you have any issue with that

Also @technotion I am curious in your RAM usage what is taking up the most percentages? Is it also System and Bitcoin?

[technotion]

@JimSB a few points of reference. My hypervisor reports a constant ~95% Memory usage (15.25GB of 16GB). Inside of the Umbrel GUI it reports, 4GB/16GB. Running an htop command, in the CLI reflects what the Umbrel GUI shows.

Running your command I see the top services:

• /app/_build/prod
• node /backend
• bitcoind
• electrs

and some others which roughly add up to the Umbrel reported 4GB. Somehow the guest OS on the Hypervisor is up allocating, reserving and using the RAM of the host server. But inside of the guest OS it tells a different story...

Proxmox 7.4.3 Ubuntu 22.04.2 Umbrel 0.5.3