Closed svewa closed 5 years ago
The last thing I am seeing on strace
is that we are polling on fds 5, 7, 17, 20, and 19.
pay
plugin. Seems to have gotten its manifest out and is now waiting for commands, so fine.autoclean
plugin. Same as above.hsmd
.connectd
.gossipd
.This poll is thus the main io-loop.
The issue is that topology should be scanning forward for new blocks continuously, but strace shows it has only scanned one block, then after reading it, topology updates the db, but suddenly we enter the main loop (which we should not, because, topology should have its own loop....?).
@cdecker there may be a race condition? The main io_loop
blocks on poll
, but the topology loop should be running?
I believe I have cracked this issue.
After starting up plugins and the core daemons (hsmd
, gossipd
, connectd
), one of the things we do is setup_topology
:
As the comment says, it does an io_loop
, which (in the absence of struct timers
) will only exit if we execute a io_break
.
Looking inside setup_topology
, we see that io_loop
is called without any struct timers
(i.e. the NULL
arguments):
Now, just prior to the above io_loop
, we see that we initiate fee estimation:
In particular, we only execute io_break
if we get our first block and we get our first fee estimation. That's this code below:
And if we look at the logs:
2019-05-28T22:08:03.684Z lightningd(3740): bitcoin-cli estimatesmartfee 2 CONSERVATIVE exited with status 1
Looks like fee estimation from bitcoin-cli
exited with status 1
.
We search for this text exited with status
and see:
And we scroll down a little and also see that, in case of such a failure, the bitcoind
system will helpfully automatically retry the command 1 second later (in particular, it will not invoke the callback inside chaintopology
yet):
Of course, it uses a timer..... which the setup_topology
loop does not enable. Remember?
TLDR: If fee estimation completely fails, we retry 1 second later. Unfortunately, the initial setup_topology
creates a io_loop
that ignores timers. So we never retry fee estimation and just end up hanging.
@cdecker @rustyrussell it seems to me that the simplest solution is to write a io_loop_with_timers
that handles timers, and merge the mainloop and the tiny io_loop
in setup_topology
to io_loop_with_timers
. Is this safe? Why does the chaintopology
io_loop
ignore timers?
https://drop.geekbox.info/feeestimateerror.log
It seems the error was the whole time, that my bitcoind was 17.1 and did not support the 2nd paramter to the getsmartfeeestimate.... your patch made the verbose error message visible, but a check if the available bitcoind has the required features or minimum version would be helpful for sure, too.
@svewa okay, I shall open a separate new issue.
It seems the error was the whole time, that my bitcoind was 17.1 and did not support the 2nd paramter to the getsmartfeeestimate.... your patch made the verbose error message visible, but a check if the available bitcoind has the required features or minimum version would be helpful for sure, too.
No, you get that because bitcoind could not estimatesmartfee
(not because of a bad JSONRPC arg), I had this issue before I don't remember how I solved it. I'll search and keep you updated.
Did you change your datetime (had a powercut or something) recently ? What are the bitcoin logs on an estimatesmartfee
?
No, you get that because bitcoind could not estimatesmartfee (not because of a bad JSONRPC arg)
Yes, I did. Just see the output in https://drop.geekbox.info/feeestimateerror.log
2019-05-31T18:22:32.564Z lightningd(10275): bitcoin-cli estimatesmartfee 2 CONSERVATIVE exited with status 1
When you run this very command bitcoin-cli estimatesmartfee 2 CONSERVATIVE
against bitcoind 0.17.1, you get the exact same error as I got. It's bitcoind telling you, that you gave the wrong parameters. If you run it against 0.18.0 it works, also the help text of bitcoind describing the estimatesmartfee
reflects that.
I have c-lightning running for over a year now, but some time maybe two month ago I noticed it stopped and would not properly start again. I tried updating to 0.7.0 and now to latest git, does not help.
config file:
starting lightningd: