Closed RequestPrivacy closed 1 year ago
After running roughly 20h just fine I have the same problem again. The connection closes but maybe its bitcoinds fault:
Mar 16 13:04:03 debian Fulcrum[1554690]: [2023-03-16 13:04:01.350] <BitcoinD.3> TCP BitcoinD.3 (id: 4) 127.0.0.1:8332: error 1 (The remote host closed the connection)
Mar 16 13:04:03 debian Fulcrum[1554690]: [2023-03-16 13:04:02.192] <BitcoinD.3> Lost connection to bitcoind, will retry every 5 seconds ...
Mar 16 13:04:03 debian Fulcrum[1554690]: [2023-03-16 13:04:01.350] <BitcoinD.2> TCP BitcoinD.2 (id: 3) 127.0.0.1:8332: error 1 (The remote host closed the connection)
Mar 16 13:04:03 debian Fulcrum[1554690]: [2023-03-16 13:04:02.193] <SynchMempool> 762083: FAIL: bitcoind connection lost
Mar 16 13:04:03 debian Fulcrum[1554690]: [2023-03-16 13:04:01.885] <Controller> 13115 mempool txs involving 149643 addresses
Mar 16 13:04:03 debian Fulcrum[1554690]: [2023-03-16 13:04:01.680] <BitcoinD.1> TCP BitcoinD.1 (id: 2) 127.0.0.1:8332: error 1 (The remote host closed the connection)
Mar 16 13:04:03 debian Fulcrum[1554690]: [2023-03-16 13:04:02.193] <BitcoinD.2> Lost connection to bitcoind, will retry every 5 seconds ...
Mar 16 13:04:03 debian Fulcrum[1554690]: [2023-03-16 13:04:02.193] <BitcoinD.1> Lost connection to bitcoind, will retry every 5 seconds ...
Mar 16 13:04:03 debian Fulcrum[1554690]: [2023-03-16 13:04:02.193] <Controller> Failed to synch blocks and/or mempool
Mar 16 13:04:06 debian Fulcrum[1554690]: [2023-03-16 13:04:06.945] <BitcoinD.1> Unable to open cookie file '/home/bitcoin/.bitcoin/.cookie': Permission denied
Mar 16 13:04:06 debian Fulcrum[1554690]: [2023-03-16 13:04:06.945] <BitcoinD.2> Unable to open cookie file '/home/bitcoin/.bitcoin/.cookie': Permission denied
Mar 16 13:04:06 debian Fulcrum[1554690]: [2023-03-16 13:04:06.945] <BitcoinD.3> Unable to open cookie file '/home/bitcoin/.bitcoin/.cookie': Permission denied
It seems that the .cookie file has been newly created at the same time fulcrum fails to connect:
# compare to "Mar 15" of OP
pi@debian: $ ls -la | grep .cookie
-rw-r----- 1 bitcoin btcCookie 75 Mar 16 13:04 .cookie
pi@debian: $ bitcoin-cli uptime
7032 # roughly 2h which is consistent with the creation time of the cookie file
Will investigate why bitcoind shuts itself down. Any thoughts/help is appreciated though.
Going to close this off as it doesn't look like fulcrum is the problem here but handles the matter rather superb!
Closing off as not related to fulcrum but rather bitcoind.
Oh hi I just saw this, sorry for the late reply. What version of bitcoind are you running? I would be curious too if there are bugs in bitcoind. Please let me know anything you discover in this regard, and do let me know the exact version you are running so I can look at it!
Maybe also run bitcoind with -debug=all
or something (note: will have lots of log spam in bitcoind's debug.log
if you do that!!).
Thanks for the reply and no worries, the issue is only 1 day old ;)
The version is Bitcoin Core 22.0.0.
pi@debian: $ sudo journalctl -fu bitcoind --since="2023-03-16 12:00:00"
Mar 16 13:04:02 debian systemd[1]: bitcoind.service: Main process exited, code=killed, status=9/KILL
Mar 16 13:04:02 debian systemd[1]: bitcoind.service: Failed with result 'signal'.
Mar 16 13:04:02 debian systemd[1]: bitcoind.service: Consumed 1h 27min 19.471s CPU time.
Something seems to kill bitcoind. I don't think its a bug really, more a limitation of my hardware / the Pi. Decreased the memory consumption with dbcache
a bit, maybe its because the pi runs out of RAM?
Will keep you posted and try with more verbose log output.
And there it is:
# /var/log/syslog
Mar 16 13:04:02 debian kernel: [17352780.421814] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=bitcoind,pid=1659642,uid=1001
Mar 16 13:04:02 debian kernel: [17352780.422033] Out of memory: Killed process 1659642 (bitcoind) total-vm:6718536kB, anon-rss:1547428kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:10384kB oom_score_adj:0
Mar 16 13:04:02 debian kernel: [17352780.723486] oom_reaper: reaped process 1659642 (bitcoind), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sorry to have spammed you with the issue. And thanks for such a fantastic and performant indexer!
Hmm that's strange. Yeah a smaller -dbcache
can help. But also if you are using jemalloc
(or I forget if bitcoind is linked with jemalloc), it can cause the process to hang on to memory for wayyy too long and not release it to the OS. Things to try:
MALLOC_CONF
to instruct jemalloc to not use tcache
. Put the following in the ENV to the process: MALLOC_CONF=tcache:false
-dbcache
may help on memory-constrained systems.. but I actually suspect the allocator is not releasing memory to the OS in a timely manner so I would try the MALLOC_CONF
thing first.Oh I guess Bitcoin Core doesn't use jemalloc, my bad. But yes -- in glibc's mallox there is something similar and yes you should DEFINITELY turn it to MALLOC_ARENA_MAX=1
. To do that, in your bitcoind.service
you should have a line before ExecStart
in your [Service]
section such as: Environment="MALLOC_ARENA_MAX=1"
.
See this link: https://www.baeldung.com/linux/systemd-services-environment-variables#setting-variables-with-environment
Thanks for providing the instructions. After setting things up it seems that the RPi is stable again - without its memory running away.
Thanks a lot for your help!
My pleasure!
Note - investigation revealed this issue isn't related to fulcrum
Description
I run a bitcoind (Bitcoin Core v22.0.0) and fulcrum (1.7.0) on the same Pi 4 and use cookie authentication.
To keep permission to an absolute minimum I allow read access to the .cookie file to a dedicated btcCookie group in which the fulcrum user is a member:
To achieve the above group setting I use
-startupnotify
in mybitcoind.service
fileDuring start-up fulcrum has no problem to connect to bitcoind.
Problem
Sometimes fulcrum looses connection to bitcoind. I'm not yet sure why.
So my question is: a.) what could be the cause that the connection is lost in the first place? The pi is only running bitcoind, fulcrum and sparrow server (a wallet with minimal terminal GUI). Load averages 3.05G RAM of the total 3.7G available, no swap. b.) what's the matter with the cookie-file (fulcrum is denied access but after a while re-allowed to read the .cookie?)
Regarding a.) is there a setting to minimize fulcrums load on the RAM? Maybe a spike in load is causing the issue for fulcrum (and bitcoind for that matter)?