Allow basic RPC handling during initial sync

[KayBeSee]

Hey Calin,

Unless I am doing something wrong, I am noticing that Fulcrum rejects all requests while it is doing its initial block sync. When I try to connect, I get connect errror: connect ECONNREFUSED 172.18.0.3:50001.

I think it would be nice from a user experience / developer experience stand point to accept basic RPC calls during initial sync so that a web UI that relies on Fulcrum can communicate sync progress to the user. Basically, just what block Fulcrum is currently synced up to.

I'm not sure if it's helpful, but Electrs implemented a similar feature a few months ago. Here is the PR for it: https://github.com/romanz/electrs/pull/543

If there is a way of doing this already, please let me know!

[cculianu]

There is a way of seeing the progress of Fulcrum by hitting the /stats endpoint. In the conf file, you define a stats = 0.0.0.0:8080 for example (here it will bind to all interfaces on 8080, and then you can hit it with http://xx.yy.zz.ww:8080/stats with a browser. Note that xx.yy.zz.ww will be the IP address of the server. You will get a bunch of JSON nonsense but buried in there I believe is some progress info. It's a LOT of data you get there. This /stats endpoint is really designed for integration into a monitoring tool.

As for generic RPC support for the server to see progress -- oh you mean the FulcrumAdmin port? Yeah, that doesn't start until the server is first synched. The reason being basically i didn't think that would be useful to anybody to admin a server that has 0 clients connected, since most of the admin commands concern themselves with kicking users, etc.

The intended use-case for Fulcrum was and is for an administrator to have access to the server's output (stdout, stderr and/or system logs) as it is synching. That's where the synch progress goes. Synching is a 1-time affair and the idea was the admin should monitor it, as they would any other setup they are doing. The output, therefore, is the primary place to see the synch progress. If Fulcrum was started by systemd, for instance, you can use journalctl to view the logs in real-time to see the synch progress.

As for the other RPC ports, the Electrum 1.4 protocol server port(s) intentionally aren't started until after first synch since it doesn't make much sense to have a stale public-facing wallet server with e.g. the latest tip from 2012. For the health of the Electrum network it intentionally does not open the Electrum 1.4 ports until it is on a recent chain tip.

I hope this helps.

[cculianu]

tl;dr: The above is a long-winded way of saying: "Duude, just use tail -f and/or journalctl -f !!!" 😄

[lukechilds]

Hey @cculianu, very impressive work on Fulrcum, I'm excited to try it out!

Just to add some more context to this issue, the use case would be running Fulcrum for a personal user who is non-technical. They just one click install the service on their personal @getumbrel server and get a connection string to connect their wallets. So they wouldn't be likely to need to kick/ban peers since it would only be themselves connecting.

However what would be useful is to view the sync progress so they have an idea for how long they need to wait before they are able to setup their wallets. Dumping the raw Fulcrum logs in the UI would probably be a bit too verbose since many of the intended users are non-technical, we just need a way to get the index % from Fulcrum and show that to the user in a UI layer with a simple progress bar.

So looks like currently the available options for that are:

• Parse sync process from Fulcrum stdout stream
• Poll /stats endpoint and grab the value from there

To me it looks like the /stats endpoint is the best fit, is there a reason you think that's not a good idea and we should instead process stdout knowing the full context of our use case?

[cculianu]

Parse sync process from Fulcrum stdout stream

Hmm. Litertally the only thing Fulcrum spits out to stdout as it's synching is synch progress. It's literally all there is. It's not much to parse.. you just read it....

Oh you are talking about doing this for an automated system so you integrate fulcrum into a UI? I see.

Yes, /stats likely is the best candidate. Somewhere buried in the JSON is a section called "Controller" and it should have stats on synch progress. If that is not the case it can be added. For an automated system I do think that is the best bet since it's structured data already (JSON) and it either already has the info you need (or it can be easily added if I forgot to add it). I am getting ready to get on a plane to Europe now so I don't have time to check the code but I'm like 75% sure I did add synch progress to /stats. It's been a few years and I added everything but the kitchen sink to stats, so I don't remember specifically if synch progress was added, but I would be surprised if I didn't add it since when I developed Fulcrum I did use /stats to keep tabs on what it's doing using structured data...

[lukechilds]

Oh you are talking about doing this for an automated system so you integrate fulcrum into a UI? I see.

Yes exactly!

Yes, /stats likely is the best candidate.

Got it, thanks for the clarification, have a safe flight!

[cculianu]

Thank you! And do let me kow if /stats has that info -- I distinctly remember hitting stats over and over again as I was devving and seeing it there. But maybe I took it out or something. I really threw everything into stats which is why I forget. Let me know if it needs to be added or if you can't find it. (Like I said I have to check the code as to what's there and I am busy packing still, ha ha).