Noob alert - Multiple issues when syncing to bitcoind

[slimscribs]

Context: I'll preface this with a mea culpa - I'm a massive noob trying to learn some tech skills, have made it thus far following k3tan's excellent series, & I've hit a road block which I probably erected...

At this point I don't know where to start troubleshooting, if anyone can provide some guidance or pointers I'd be super grateful!

System: I'm running bitcoin-24.0.1 on ubuntu 20.04.6, & have installed fulcrum-1.9.0 downloaded. Hardware wise: 2T SSD & 8 GB RAM.

Issue: When I run fulcrum.service it almost immediately fails to progress. Below is a snippet of the related logs, there are many many rows of information I don't understand but a number of lines jump out which could give a clue to the problem. I just have no idea what's relevant, which to prioritise or where/how to start solving.

Any ideas?

Thanks in advance!!

--unexpected content type / error processing

--lost connection to bitcoind

--authentication failed

[cculianu]

Probably the most likely reason for the failure is that the name/password for bitcoind don't match. Does it ever succeed in connecting to bitcoind?

Are you using rpccookie auth or regular rpc auth? Sorry I'm super experienced here with bitcoind (and Fulcrum, of course) so it's hard for me to empathize to the knowledge level of a noob. I will continue to issue high level commands and questions, realizing on some level that someone that is new may not know what I'm talking about.

Let me try and break it down.

Do you have both an rpcuser=<some username> and rpcpassword=<some string> in your main bitcoin.conf for bitcoind?

If you have both of those, you are using password authentication. Make sure that in the fulcrum.conf you have matching rpcuser= and rpcpassword= lines.

If you only have an rpcuser= line, (or no rpcuser= line), then you are using "cookie based" authentication. In that case, fulcrum.conf needs to know the location of the RPC cookie file that bitcoind generates each time. Usually it would be something like rpccookie=/path/to/somebitcoinddatadir/datadir/.cookie

Does that help?

[PlebEDU]

I'm having similar issues following the same series and also a beginner. Looking at my bitcoin.conf there is rpcauth=bitcoin:[output of rpcauth.py] as the guide we are following has us use rpcauth.py so is that "cookie based" authentication?

Thanks for your help @cculianu

[slimscribs]

Thanks for coming back so quickly, & for making the effort to break it down.

No, fulcrum.service never succeeds in connecting to bitcoind (I've tried a handful of times since April without luck).

Exactly as @PlebEDU notes, the stripped down bitcoin.conf has rpcauth=, there is no reference to rpcuser= or rpcpassword=. However, the same guide does uncomment / include rpcuser= & rpcpassword= in fulcrum.conf. So no obvious (to me) name/password to match under the current setup?

If cookie authentication is the issue to troubleshoot around, is it simply a case of adding the pathway to the fulcrum.conf?

Btw here's the structure of bitcoin.conf, incase anything looks awry: server=1 txindex=1 daemon=1 rpcport=8332 rpcbind=0.0.0.0 rpcallowip=127.0.0.1 rpcallowip=10.0.0.0/8 rpcallowip=172.0.0.0/8 rpcallowip=192.0.0.0/8 zmqpubrawblock=tcp://0.0.0.0:28332 zmqpubrawtx=tcp://0.0.0.0:28333 zmqpubhashblock=tcp://0.0.0.0:28334 whitelist=127.0.0.1 rpcauth=bitcoin:[output of rpcauth.py] proxy=127.0.0.1:9050 listen=1 bind=127.0.0.1 onlynet=onion

[cculianu]

Ok so it looks like you are using the rpcauth mechanism which is an obfuscated way to do username/password auth, really. It's equivalent to rpcuser=bitocoin and rpcpassword=<cleartext_pw>, except with rpc_auth, the password is stored in the conf file as a salted hash.

What you should do, then, is in fulcrum.conf, put:

rpcuser=bitcoin  # Or whatever username you picked, usually default is bitcoin for most nub ppl :)
rpcpassword=<cleartext_pw_you_used_as_input_to_rpcauth.py>

That is, just have the cleartext password sitting around in fulcrum.conf. There is no other way to do it, because the client, fulcrum, does indeed need to know the real password in order to log-in (the server doesn't need to know it it just needs the password hash to authenticate a client).

I hope that is clear?

[slimscribs]

Crystal clear. Yep, my fulcrum.conf already had rpcuser=bitcoin (hehe) & rpcpassword=cleartext prior to raising the issue... :'(

I'm going to try removing rpcauth= and replacing with rpcuser=bitcoin & rpcpassword=cleartext per fulcrum.config within bitcoin.config... hopefully that works.

If that fails, will I stuff anything up if I just delete all fulcrum related directories & files, & try starting from scratch but avoid setting up with rpcauth.py?

[cculianu]

If that fails, will I stuff anything up if I just delete all fulcrum related directories & files, & try starting from scratch but avoid setting up with rpcauth.py?

I'm not sure if that will really help but since you never really "synched" it may...?

[slimscribs]

Hmm still no luck. I've updated bitcoin.conf by replacingrpcauth= with rpcuser=bitcoin & rpcpassword=cleartext taken directly from fulcrum.conf so I'm 100% confident they align...

When I run sudo systemctl status fulcrum.service this is the fine line of the log:

Oct 09 08:57:00 nodebox Fulcrum[772]: [2023-10-09 08:57:00.054] Got unexpected content type: text/html; charset=ISO-8859-1; will log the rest of this HTTP response

Then when I run the full log its immediately back to "lost connection to bitcoind", followed by some reference to "failure to parse json from string", & then on "authentication to bitcoind rpc failed. Please check the rpcuser & rpcpass are correct & restart".

If this looks like it's still just user error let me know & I'll try to figure out another approach rather than bugging you.

Cheers

Oct 09 09:01:55 nodebox Fulcrum[772]: [2023-10-09 09:01:55.051] Error processing data coming in: Failed to parse Json from string: Oct 09 09:01:55 nodebox Fulcrum[772]: [2023-10-09 09:01:55.051] Lost connection to bitcoind, will retry every 5 seconds ... Oct 09 09:01:59 nodebox Fulcrum[772]: [2023-10-09 09:01:59.799] Waiting for bitcoind... Oct 09 09:02:00 nodebox Fulcrum[772]: [2023-10-09 09:02:00.050] Got HTTP status 401 Unauthorized; will log the rest of this HTTP response Oct 09 09:02:00 nodebox Fulcrum[772]: [2023-10-09 09:02:00.050] Got HTTP status 401 Unauthorized; will log the rest of this HTTP response Oct 09 09:02:00 nodebox Fulcrum[772]: [2023-10-09 09:02:00.050] Authentication to bitcoind rpc failed. Please check the rpcuser and rpcpass are correct and restart! Oct 09 09:02:00 nodebox Fulcrum[772]: [2023-10-09 09:02:00.050] 401 (header): WWW-Authenticate: Basic realm="jsonrpc"

[EchterAgo]

Have you checked the bitcoind log for errors?

[slimscribs]

@EchterAgo I had not... thanks for the suggestion, there is indeed an issue which looks related to my fulcrum problem:

~/.bitcoin$ tail -f ~/.bitcoin/debug.log 2023-10-09T21:29:34Z ThreadRPCServer incorrect password attempt from 127.0.0.1:50012 2023-10-09T21:29:39Z ThreadRPCServer incorrect password attempt from 127.0.0.1:33596 2023-10-09T21:29:39Z ThreadRPCServer incorrect password attempt from 127.0.0.1:33588 2023-10-09T21:29:39Z ThreadRPCServer incorrect password attempt from 127.0.0.1:33602 2023-10-09T21:29:44Z ThreadRPCServer incorrect password attempt from 127.0.0.1:33636

With help from chatGPT I've been trying to isolate what is attempting to connect with incorrect details but haven't found the source yet.

As far as I can tell, bitcoind is working because getblockchaininfo successfully returns the current block height (811424) verified against mempool.space, which I interpret as meaning rpcuser & rpcpassword are correct, otherwise this would fail?

[EchterAgo]

That is likely the connection from Fulcrum. To me that seems like there is an authentication problem. The rest of your configuration seems fine because the programs can talk to each other.

Can you please check the bitcoind log for any RPC errors / warnings when the demon is starting? You could do a grep -i -e warning -e error -e RPC ~/.bitcoin/debug.log and share that with me, privately too, or redact any info you don't want shared and post it publicly.

[EchterAgo]

Also, do you mind sharing both bitcoind and Fulcrum configs authentication options? I mean actual values so we can check. You can make a new config with a random password.

[slimscribs]

@EchterAgo thanks so much. I'll send the details to the email address on your profile.

[cculianu]

From the symptoms described it just sounds like username/password mismatch of some sort. Or some other access restriction imposed by bitcoind somehow. Yes, I think sending Axel the configs will likely solve this .. hopefully it's something minor & easily fixed from conf files.

[EchterAgo]

rpcuser = bitcoin

So, I think you got the configuration file format for bitcoin.conf wrong. You are not supposed to put spaces around the equals sign (=) in there. If you remove the spaces it might just work.

[EchterAgo]

taken directly from fulcrum.conf so I'm 100% confident they align

I think this is how you got it wrong. fulcrum.conf and bitcoin.conf use different configuration file parsers, so there are subtle differences.

[EchterAgo]

As for the rpcauth.py issue, can you please share the actual setting you use in fulcrum.conf and bitcoin.conf for rpcpassword? I mean the full line, nothing redacted. It is just a password, you can use a random one and generate the values, and when finished diagnosing choose a new password.

[slimscribs]

You are not supposed to put spaces around the equals sign (=) in there. If you remove the spaces it might just work.

This is 100% the issue.

After removing the spaces in bitcoin.conf & restarting fulcrum.service the connection is successfully made & is currently 27.1% complete. Something so simple.

Thank you so much, really appreciate your help as well as @cculianu.

I've replied to your email with rpcauth.py details. The successful bitcoin.config file does not use rpcauth.py, but instead rpcuser= &rpcpassword=.

Is it worth replacing these with the rpcauth.py approach instead?

If yes, does one configure the files using the following authentication details: bitcoin.conf rpcauth=bitcoin:[output of rpcauth.py]

fulcrum.conf rpcuser = bitcoin rpcpassword = cleartextpassword

OR should fulcrum.conf include rpcauth:[output of rpcauth.py]