After the Last upgrade to ver 0.3.18 or may be one before myNode constantly crashes within 15-20 hrs

[h33fn]

Describe the bug After the Last upgrade to ver 0.3.18 or maybe one before myNode constantly crashes within 15-20 hrs, otherwise first 10-15 hrs no issues using it and no errors, and everything is as expected, but eventually VM aborting this image (see error logs attached ) My current myNode 0.3.18

Expected behavior In the past, myNode was able to run for months with VM in the background until I will reboot it for maintenance, and currently, I don't see any crashes/errors on my server hardware which is ASUSTOR AS6602T Intel® Celeron™ CPU @ 2.00GHz 8.00 GB and no Disk Errors reported

Screenshots

Desktop (please complete the following information):

• Browser chrome
• Version MacOS

MyNode hardware (please complete the following information):

• Device: [VM]
• Version [Virtual Box 6.1.32 (149290)
• ASUSTOR AS6602T Intel® Celeron™ CPU @ 2.00GHz 8.00 GB

Additional context Attaching crash logs from VM VBox.log 1.txt VBox.log 2.txt VBox.log 3.txt VBox.log.txt

[tehelsper]

It looks like it shuts off intentionally shortly after a window resize. If you log a support ticket, we can take a further look at the logs, but it doesn't appear to be a MyNode software issue. I'd guess RAM or memory problem. There was a recent LND bug reported about a memory leak.

[onlineapps-cloud]

hi, your device is power off? i correctly understand? if yes, i had same situation with my Raspberry pi, in 1-2 or 3 days, my raspberry shutting down, or other crash type, because in monitor i don't have video signal, i need to restart, after few days situation happens again, i thinked that my hardware have problems, but if you had same issue on different hardware, i think problem is not in hardware

[h33fn]

It looks like it shuts off intentionally shortly after a window resize. If you log a support ticket, we can take a further look at the logs, but it doesn't appear to be a MyNode software issue. I'd guess RAM or memory problem. There was a recent LND bug reported about a memory leak.

I'm running server and its running solid with no critical errors https://www.youtube.com/watch?v=FX6yMC74QJM

[h33fn]

hi, your device is power off? i correctly understand? if yes, i had same situation with my Raspberry pi, in 1-2 or 3 days, my raspberry shutting down, or other crash type, because in monitor i don't have video signal, i need to restart, after few days situation happens again, i thinked that my hardware have problems, but if you had same issue on different hardware, i think problem is not in hardware

its a server based hardware above asustor storage server device and i'm running Virtual Box within a server and VB acts like device and yes myNode environment within VBox shuts down within 10-15 hrs after successful restart.

[h33fn]

hi, your device is power off? i correctly understand? if yes, i had same situation with my Raspberry pi, in 1-2 or 3 days, my raspberry shutting down, or other crash type, because in monitor i don't have video signal, i need to restart, after few days situation happens again, i thinked that my hardware have problems, but if you had same issue on different hardware, i think problem is not in hardware

I believe it is a memory leak, my configuration never changed and amount of apps running under my node exactly the same for the past 12+ months

error log from myNode status page

Low MemoryYour device was running low on memory and a process has been restarted. You may want to disable some applications.[ 1833.364551] Out of memory: Killed process 12986 (netdata) total-vm:89892kB, anon-rss:25808kB, file-rss:0kB, shmem-rss:500kB, UID:201 pgtables:160kB oom_score_adj:1000 --

[onlineapps-cloud]

hi, i installed few days ago mynodebtc on clean debian, using this tutorial https://github.com/mynodebtc/mynode#install-on-existing-system-beta today, mynodebtc stuck, ping working, ssh not working, gui (openbox) not working, just black screen, i have 4 cpu i5 10 gen, 6 GB ram, problem is not in hardware, i had same problem on my raspberry. mynodebtc stuck on full bitcoin sync ~ 32%

[tehelsper]

You would need to check your Bitcoin peers to see why the sync stopped.

For RAM usage, there is an LND update that fixes a memory leak. It will be in the next release.

[sicbitm]

Yes!! same problem here, with mynode on a VM on proxmox. The VM freeze with CPU at 100% every 24 approx

This is the only log I got to catch:

[onlineapps-cloud]

You would need to check your Bitcoin peers to see why the sync stopped.

For RAM usage, there is an LND update that fixes a memory leak. It will be in the next release.

Last release don't fix problem, i can confirm, exist memory leak i have this apps installed only:

memory leak proof: mynodebtc hardware after update to 0.3.19 after ~24 hours after ~24 hours after ~24 hours i get unexpected system crash. RAM 4GB. SWAP 3GB.

[h33fn]

yes the last update 0.3.19 did nothing and actually myNode stays active for about 4-8 hrs before crashing used to be 8-15hrs

[onlineapps-cloud]

glances info

[onlineapps-cloud]

2023-07-22 15:57:23 (00:07:36) - CRITICAL on MEM (96.1)

[tehelsper]

@h33fn Could be caused by the update to BTC Pay server using more memory.

@onlineapps-cloud You definitely need more RAM than 4GB if you want to run BTCPay and Mempool at the same time.

In general, watch Glances like in the above screenshots, and if you see memory warning add more RAM.

[h33fn]

BTC Pay Server was off , I just turned Mempool off , let me see if it will crash or not

[onlineapps-cloud]

now i have running only this apps: but today i had unexpected crash again resource usage:

[h33fn]

BTC Pay Server was off , I just turned Mempool off , let me see if it will crash or not

I removed BTC pay and mempool , restarted and everything was ok. Now I'm running Mempool only without BTC, and everything is fine after the last update. Thanks for your support.

[cexshun]

Been happening to me too, currently on 0.3.19. Increasing the swap file to 4GB has stopped it from crashing, but RAM usage currently sitting at 90% and swap usage at 40%. Running core, lightning, electrum, mempool, and explorer on a Pi4 4GB. Bitcoind sitting at over 5GB virtual and over 1GB res. NodeJS is also over 1GB res memory. I've also attempted to limit bitcoind memory usage with considerable limitation to the config without much result.

dbcache=250 maxorphantx=10 maxmempool=200 maxconnections=20 maxuploadtarget=1000 rpcworkqueue=32 mempoolfullrbf=1

[tehelsper]

That's good to know - perhaps new versions of BTC Pay server are using more RAM than before. Looks like a 4GB device may not be enough to keep up with that app.

[cexshun]

That's good to know - perhaps new versions of BTC Pay server are using more RAM than before. Looks like a 4GB device may not be enough to keep up with that app.

I'm not now, nor ever have had BTC Pay enabled. So at least in my case, that's not the culprit. But I'm agreeing with the end conclusion, maybe 4GB just isn't enough to run the mynodebtc suite any longer. I was going to install node-exporter and cadvisor to try and monitor the resources, but the added resource draw for these services just isn't sustainable.

[tehelsper]

@cexshun Your case sounds like its btc rpc explorer (it uses nodejs).

[cexshun]

Alright, disabled and restarting just for debugging purposes. I likely won't leave it disabled since it runs without crashing with the 4GB swap file, but it seems like valuable information to have.

[cexshun]

node /backend/package/index.js

Even with RPC explorer disabled, that process still munches memory like mad. Second only to bitcoind.

[tehelsper]

You can run this to possibly find which service that instance of node belong to.

sudo systemctl status < pid >

[cexshun]

c191868384ff mempool/backend:v2.5.0 "docker-entrypoint.s…" 4 hours ago Up 4 hours 8999/tcp mempool_api_1

mempool_api docker container

[tehelsper]

Sorry, github hid part of that command. Enter the process id of the node process to find the app.

[cexshun]

That process is running under a docker container. Checking docker, the containerID is for mempool_api_1.