Continuously "Waiting for bitcoind..."

[sutt0n]

Hello,

When attempting to connect to the bitcoind in my setup, I continuously see Waiting for bitcoind.... I've verified my credentials and connection config are all correct, and I can ping / resolve the instance just fine while SSH'd into my Fulcrum machine.

Here are the logs I'm receiving:

2024-04-06 08:01:54.446 CDT
[2024-04-06 13:01:54.446] BitcoinDMgr: starting 3 bitcoin RPC clients ...
[2024-04-06 13:01:54.446] BitcoinDMgr: started ok
[2024-04-06 13:01:54.447] Stats HTTP: starting 1 server ...
[2024-04-06 13:01:54.447] Starting listener service for HttpSrv 0.0.0.0:8080 ...
[2024-04-06 13:01:54.448] Service started, listening for connections on 0.0.0.0:8080
[2024-04-06 13:02:04.239] <Controller> Waiting for bitcoind...
[2024-04-06 13:02:14.239] <Controller> Waiting for bitcoind...
[2024-04-06 13:02:24.239] <Controller> Waiting for bitcoind...
[2024-04-06 13:02:34.239] <Controller> Waiting for bitcoind...

Here's my fulcrum.conf:

rpcpassword=<redacted>
tcp = 0.0.0.0:50001
ssl = 0.0.0.0:50002
ws = 0.0.0.0:50003
wss = 0.0.0.0:50004
stats = 0.0.0.0:8080
bitcoind = bitcoind.<namespace>.svc.cluster.local:8332
key = /.fulcrum/tls.key
cert = /.fulcrum/tls.cert
rpcuser = rpcuser
datadir = /.fulcrum/db
bitcoind_clients = 3
peering = false
announce = false
fast-sync = 1024
worker_threads = 0
bitcoind_throttle = 25 10 2

I'm sure it's something silly or simple that I'm missing. Thank you in advance!

[sutt0n]

For what it's worth, here are debug logs:

[2024-04-07 23:44:32.238] <Controller> Waiting for bitcoind...
[2024-04-07 23:44:37.234] <BitcoinD.1> (Debug) TCP BitcoinD.1 (id: 2) reconnecting...
[2024-04-07 23:44:37.234] <BitcoinD.1> (Debug) TCP BitcoinD.1 (id: 2) socket state: 0
[2024-04-07 23:44:37.234] <BitcoinD.1> (Debug) TCP BitcoinD.1 (id: 2) socket state: 1
[2024-04-07 23:44:37.234] <BitcoinD.1> (Debug) TCP BitcoinD.1 (id: 2) socket state: 2
[2024-04-07 23:44:42.237] <BitcoinD.1> (Debug) TCP BitcoinD.1 (id: 2) reconnecting...
[2024-04-07 23:44:42.237] <BitcoinD.1> (Debug) TCP BitcoinD.1 (id: 2) socket state: 0
[2024-04-07 23:44:42.237] <BitcoinD.1> (Debug) TCP BitcoinD.1 (id: 2) socket state: 1
[2024-04-07 23:44:42.237] <BitcoinD.1> (Debug) TCP BitcoinD.1 (id: 2) socket state: 2
[2024-04-07 23:44:42.239] <Controller> Waiting for bitcoind...

Note: I have other services successfully connecting and interacting with my bitcoind node via RPC with no problem.

[cculianu]

1. What Fulcrum version?
2. Can you telnet to the 8332 port on that bitcoind machine just fine?
3. I think by default the RPC service on bitcoin core doesn't allow anything but localhost so be sure your bitcoin.conf config is set up to allow on-localhost.

[sutt0n]

1. What Fulcrum version?
2. Can you telnet to the 8332 port on that bitcoind machine just fine?
3. I think by default the RPC service on bitcoin core doesn't allow anything but localhost so be sure your bitcoin.conf config is set up to allow on-localhost.

1. 1.9.7
2. I can telnet to it, am prompted for a password, and then it hangs.
3. For testing purposes for now, this is a forked staging environment with the config for RPC IPs set to rpcallowip=0.0.0.0/0.

[sutt0n]

FWIW, my Bitcoin Core version is 24.0.1, and here is my bitcoin.conf:

rpcpassword=<redacted>
debug=mempool
debug=rpc
server=1
txindex=1
printtoconsole=1
rpcuser=rpcuser
zmqpubrawtx=tcp://0.0.0.0:28333
zmqpubrawblock=tcp://0.0.0.0:28332
blockfilterindex=1
rpcworkqueue=256
rpcport=8332
assumevalid=00000000000000000001e73b1c55118c6b7b3874870253e59d195203354763a5
dbcache=1200
bind=0.0.0.0
rpcallowip=0.0.0.0/0
rpcbind=0.0.0.0
mainnet=1
[test]
bind=0.0.0.0
rpcallowip=0.0.0.0/0
rpcbind=0.0.0.0
[regtest]
bind=0.0.0.0
rpcallowip=0.0.0.0/0
rpcbind=0.0.0.0
[signet]
bind=0.0.0.0
rpcallowip=0.0.0.0/0
rpcbind=0.0.0.0

For more context, this is deployed in a private K8s cluster. I have a LND node that also talks to bitcoind via RPC, and I have VPC logs that show dest/src from LND -> bitcoind that shows both packets and bytes sent/recvd; however, I have only packets sent via Fulcrum -> bitcoind and 0 bytes. 😕

[cculianu]

But lnd node is running on same box and Fulcrum on a different box, correct?

IDK to me it looks like communication trouble between Fulcrum -> bitcoind unrelated to Fulcrum itself and either a blocker is in bitcoind or from your network setup.

I suggest going to the same box that Fulcrum is on, dropping into a shell, and doing: telnet ip.or.hostname.of.your.bitcoind 8332 and seeing if it connects and works or not. Maybe send it a fake http request, etc.

Also note that bitcoin core now differentiates more strongly between ipv4 and ipv6. See this thread: https://github.com/bitcoin/bitcoin/issues/21070

You may need to also do rpcallowip=::/0 and rpcbind=[::] (I actually forget the exact syntax for ipv6) in bitcoin.conf... if for example your Fulcrum box is attempting to connect via ipv6 only (which it may be doing seeing as how you specified a hostname in your fulcrum.conf and not an IP address).

Anyway good luck.

[sutt0n]

This was fixed, and had to do with a label-based NetworkPolicy that I completely missed in my environment.

My apologies for this oversight and thank you for your help regardless!

[cculianu]

Eh, it happens. Glad you figured it out.