clightning4j / btcli4j

It is a core lightning plugin to override Bitcoin backend plugin with esplora by Blockstream and give the possibility to make the running process with bitcoind in pruning mode more solid.
GNU General Public License v2.0
8 stars 7 forks source link

Crashing in pruned mode when rpc configured #46

Closed sha-265 closed 3 years ago

sha-265 commented 3 years ago

When I'm setting up the parameters bitcoin-rpcuser, bitcoin-rpcpassword & bitcoin-rpcurl in .lightning/config file, lightningd is crashing with this error:

2021-07-26T11:20:47.456Z DEBUG plugin-btcli4j-gen.sh: GetChainInfoCommand: GetChainInfoCommand: Block count = 692740 java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at jrpc.clightning.plugins.rpcmethods.RPCMethodReflection.doRun(RPCMethodReflection.java:36) at jrpc.clightning.plugins.CLightningPlugin.doMethods(CLightningPlugin.java:290) at jrpc.clightning.plugins.CLightningPlugin.start(CLightningPlugin.java:128) at io.vincenzopalazzo.btcli4j.AppKt.main(App.kt:26) at io.vincenzopalazzo.btcli4j.AppKt.main(App.kt) Caused by: java.lang.IllegalArgumentException: Arguments are/is null at io.github.clightning4j.litebtc.utils.gson.JsonConverter.deserialization(JsonConverter.java:74) at io.github.clightning4j.litebtc.utils.okhttp.HttpFactory.makeRequest(HttpFactory.java:79) at io.github.clightning4j.litebtc.LiteBitcoinRPC.makeBitcoinRequest(LiteBitcoinRPC.java:63) at io.vincenzopalazzo.btcli4j.control.commands.btcprune.EstimateFeeBtc.run(EstimateFeeBtc.kt:53) at io.vincenzopalazzo.btcli4j.control.MediationMethod.runCommand(MediationMethod.kt:77) at io.vincenzopalazzo.btcli4j.Plugin.estimateFees(Plugin.kt:101) ... 9 more 2021-07-26T11:21:14.969Z BROKEN plugin-btcli4j-gen.sh: Message empty

Without setting this parameters, it's working fine.

Thanks.

vincenzopalazzo commented 3 years ago

Hi @sha-265,

Thanks for reporting this information. I will try on my system and I will try to have an answer and a solution for the ending of the day.

Sorry about that.

vincenzopalazzo commented 3 years ago

Can you share the bitcoin config?

Thanks :)

vincenzopalazzo commented 3 years ago

Ok, I'm back @sha-265, unfortunately, I'm not able to see this exception on my server (and this is sad), but I see that the exception is caused by a null response from the bitcoind rpc interface. In particular here

Caused by: java.lang.IllegalArgumentException: Arguments are/is null
at io.github.clightning4j.litebtc.utils.gson.JsonConverter.deserialization(JsonConverter.java:74)
at io.github.clightning4j.litebtc.utils.okhttp.HttpFactory.makeRequest(HttpFactory.java:79)
at io.github.clightning4j.litebtc.LiteBitcoinRPC.makeBitcoinRequest(LiteBitcoinRPC.java:63)
at io.vincenzopalazzo.btcli4j.control.commands.btcprune.EstimateFeeBtc.run(EstimateFeeBtc.kt:53)

This means that your bitcoind could return an empty response (without error?), to see what is the problem I need some extra details, like the following one

Sorry for that list of information requirements

sha-265 commented 3 years ago

Hi @vincenzopalazzo, thanks for helping :)

I'm using Bitcoin Core v0.21.1

Bitcoin config:

txindex=0
rpcuser=user
rpcpass=password
server=1
nolisten=1
dbcache=256
rpcallowip=192.168.1.100
prune=4096

bitcoin-cli estimatesmartfee 6 output:

$ bitcoin-cli estimatesmartfee 6
{
  "feerate": 0.00001242,
  "blocks": 6
}
vincenzopalazzo commented 3 years ago

Thanks, @sha-265,

I have no idea about the motivation for this error, unfortunately. Now I improve the error message inside the Java RPC interface for bitcoin core and throws an error with some message. All the details in this PR https://github.com/clightning4j/lite-bitcoin-rpc/pull/10

I don't know how you are using the plugin, but now you need to compile it from the source with the following command ./gradlew createRunnableScript in the root of the directory.

After that, you need to link the c-lightning (plugin=path_file in config file) with the bash file called btcli4j-gen.sh that is in the root of the repository. This file is autogenerated by the compiler phase.

After that you can start the c-lightning and see what happen.

I will expect that the plugin will continue to work, but I would like to receive you c-lightning logs if it will run for a while. However, if now you will have a exception (again), we have more information on what happens.

Required a JDK version => 8 on your system.

I hope to hear your with some good new soon.

sha-265 commented 3 years ago

@vincenzopalazzo, I'm not sure I'm using the plugin as should. The readme is not so clear about what mode should I use. I'm running a pruned Bitcoin core node, so I guessed I should use the btcli4j pruned mode. And as I said when setting the parameters it's crashing. I'm just copying the rpcuser and rpcpassword from my Bitcoin core node (maybe I'm doing something wrong here?), and running my node with this command:

lightningd --plugin=/home/me/lightning/plugins/btcli4j-gen.sh --disable-plugin bcli --btcli4j-pruned

I recompiled the plugin and this is the error I get:

2021-07-27T17:21:13.198Z BROKEN plugin-btcli4j-gen.sh: GetChainInfoBtc: terminate bitcoin core with error: Error with empty message and error code: 401 2021-07-27T17:21:42.371Z BROKEN plugin-btcli4j-gen.sh: EstimateFeeBtc: terminate bitcoin core with error: Error with empty message and error code: 401 2021-07-27T17:21:42.371Z UNUSUAL lightningd: Unable to estimate opening fees 2021-07-27T17:21:42.371Z UNUSUAL lightningd: Unable to estimate mutual_close fees 2021-07-27T17:21:42.371Z UNUSUAL lightningd: Unable to estimate unilateral_close fees 2021-07-27T17:21:42.371Z UNUSUAL lightningd: Unable to estimate delayed_to_us fees 2021-07-27T17:21:42.372Z UNUSUAL lightningd: Unable to estimate htlc_resolution fees 2021-07-27T17:21:42.372Z UNUSUAL lightningd: Unable to estimate penalty fees 2021-07-27T17:21:42.372Z UNUSUAL lightningd: Unable to estimate min_acceptable fees 2021-07-27T17:21:42.372Z UNUSUAL lightningd: Unable to estimate max_acceptable fees lightningd: FATAL SIGNAL 11 (version v0.10.0) 0x543bb3 send_backtrace common/daemon.c:39 0x543c3e crashdump common/daemon.c:52 0xb7f66d6b ??? ???:0 0x555539 bitcoin_block_blkid bitcoin/block.c:230 0x4f21ad new_block lightningd/chaintopology.c:840 0x4f2359 init_topo lightningd/chaintopology.c:927 0x4f14eb getrawblockbyheight_callback lightningd/bitcoind.c:361 0x524f8d plugin_response_handle lightningd/plugin.c:460 0x5250b7 plugin_read_json_one lightningd/plugin.c:566 0x525235 plugin_read_json lightningd/plugin.c:611 0x580a99 next_plan ccan/ccan/io/io.c:59 0x581045 do_plan ccan/ccan/io/io.c:407 0x581078 io_ready ccan/ccan/io/io.c:417 0x582cf5 io_loop ccan/ccan/io/poll.c:445 0x506ca2 io_loop_with_timers lightningd/io_loop_with_timers.c:24 0x4f4402 setup_topology lightningd/chaintopology.c:1163 0x50a65a main lightningd/lightningd.c:1016 0xb7a7ab40 ??? ???:0 0x4f0bd0 ??? ???:0 0xffffffff ??? ???:0 Segmentation fault

sha-265 commented 3 years ago

I tried to run the lightning node, with this parameters but without the plugin, and it's look like there is no bitcoin-rpcurl parameter.

lightningd: /home/me/.lightning/config line 15: bitcoin-rpcurl=http://127.0.0.1:8332: unrecognized option

I checked the documentation and indeed there is no parameter like this.

Anyway, also when remove this parameter, the lightning node is unable to connect to Bitcoin core, but working fine without setting these parameters.

Is it mandatory to set this parameters for the plugin?

EDIT: I think it is mandatory to set this parameters. When I'm running the plugin without bitcoin-rpcurl but with bitcoin-rpcuser and bitcoin-rpcpassword set, the node crashes and I get this error:

java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at jrpc.clightning.plugins.rpcmethods.RPCMethodReflection.doRun(RPCMethodReflection.java:36) at jrpc.clightning.plugins.CLightningPlugin.doMethods(CLightningPlugin.java:290) at jrpc.clightning.plugins.CLightningPlugin.start(CLightningPlugin.java:128) at io.vincenzopalazzo.btcli4j.AppKt.main(App.kt:26) at io.vincenzopalazzo.btcli4j.AppKt.main(App.kt) Caused by: java.lang.IllegalArgumentException: Expected URL scheme 'http' or 'https' but no colon was found at okhttp3.HttpUrl$Builder.parse$okhttp(HttpUrl.kt:1260) at okhttp3.HttpUrl$Companion.get(HttpUrl.kt:1633) at okhttp3.Request$Builder.url(Request.kt:184) at io.github.clightning4j.litebtc.utils.okhttp.HttpFactory.makeRequest(HttpFactory.java:65) at io.github.clightning4j.litebtc.LiteBitcoinRPC.makeBitcoinRequest(LiteBitcoinRPC.java:63) at io.vincenzopalazzo.btcli4j.control.commands.btcprune.EstimateFeeBtc.run(EstimateFeeBtc.kt:53) at io.vincenzopalazzo.btcli4j.control.MediationMethod.runCommand(MediationMethod.kt:77) at io.vincenzopalazzo.btcli4j.Plugin.estimateFees(Plugin.kt:101) ... 9 more 2021-07-27T18:37:08.801Z BROKEN plugin-btcli4j-gen.sh: Message empty /home/me/lightning/plugins/btcli4j-gen.sh error: bad response to estimatefees (missing 'opening' field), response was {"id":10,"jsonrpc":"2.0","result":{}}

vincenzopalazzo commented 3 years ago

Hi @sha-265,

Sorry for the delay but I took my first shot of the vaccine and I'm a little bit tired. However, I will try to answer your question in order

I'm not sure I'm using the plugin as should. The readme is not so clear about what mode should I use.

I apologize if the Readme is unclear, I will try to improve it. I'm a bad person into write readme :) sorry for that.

I'm running a pruned Bitcoin core node, so I guessed I should use the btcli4j pruned mode.

yes you need to specify the flag --btcli4j-pruned to tell the plugin that need to run the pruning mode.

I recompiled the plugin and this is the error I get:

Thanks to recompile the version of the plugin with my last fix. Now we have the following error that get information about the error code

2021-07-27T17:21:13.198Z BROKEN plugin-btcli4j-gen.sh: GetChainInfoBtc: terminate bitcoin core with error: Error with empty message and error code: 401
2021-07-27T17:21:42.371Z BROKEN plugin-btcli4j-gen.sh: EstimateFeeBtc: terminate bitcoin core with error: Error with empty message and error code: 40

From a previous experience, the error 401 is an authentication error for bitcoin core, and this is one thread on bitcoin stack exchange https://bitcoin.stackexchange.com/questions/36969/jsonrpcclient-php-states-its-served-a-401-error-response/85046

Maybe if you run the curl request from your system we can see if it is an authentication problem. Maybe you can report the result of the following command

curl --user myusername --data-binary '{"jsonrpc": "1.0", "id": "curltest", "method": "getblockchaininfo", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:8332/

ps: Maybe on your bitcoin node do you have a cookie authentication?

I tried to run the lightning node, with this parameters but without the plugin, and it's look like there is no bitcoin-rpcurl parameter.

I try to reuse the propriety that c-lightning needs already on the config file to talk with the bitcoin core. however, the plugin it is not able to parse the file all ready to take all the info alone, so you need to specify the also the propriety bitcoin-rpcurl.

EDIT: I think it is mandatory to set this parameters. When I'm running the plugin without bitcoin-rpcurl but with bitcoin-rpcuser and bitcoin-rpcpassword set, the node crashes and I get this error:

If you set the--btcli4j-pruned the proprieties are mandatory. However, if you remove the flag you can ran the Rest API mode, without bitcoin core but with esplora API.

I hope I touch all points and I will wait to receive the feedback about the curl command.

Thanks and sorry if I have not a solution yet for you.

sha-265 commented 3 years ago

Hi @vincenzopalazzo,

I apologize if the Readme is unclear, I will try to improve it. I'm a bad person into write readme :) sorry for that.

No worries :) So pruned mode should be use if I have a pruned node and I want btcli4j to complete missing block etc? This is what I wanted.

From a previous experience, the error 401 is an authentication error for bitcoin core, and this is one thread on bitcoin stack exchange https://bitcoin.stackexchange.com/questions/36969/jsonrpcclient-php-states-its-served-a-401-error-response/85046

There was an authentication error, but I fixed it. And I think I figured out the problem with bitcoin-rcpurl set. It should only be set when the plugin is enabled. If you set the this parameter and running c-lightning without the plugin, c-lightning won't run.

Now I think it's working OK, but I see different error:

2021-07-29T16:43:24.358Z BROKEN plugin-btcli4j-gen.sh: io.github.clightning4j.litebtc.exceptions.LiteBitcoinRPCException: java.io.EOFException: \n not found: limit=0 content=? 2021-07-29T16:43:24.362Z BROKEN plugin-btcli4j-gen.sh: \tat io.github.clightning4j.litebtc.LiteBitcoinRPC.makeBitcoinRequest(LiteBitcoinRPC.java:67) 2021-07-29T16:43:24.366Z BROKEN plugin-btcli4j-gen.sh: \tat io.vincenzopalazzo.btcli4j.control.commands.btcprune.GetRawBlockByHeightBtc.run(GetRawBlockByHeightBtc.kt:50) 2021-07-29T16:43:24.370Z BROKEN plugin-btcli4j-gen.sh: \tat io.vincenzopalazzo.btcli4j.control.MediationMethod.runCommand(MediationMethod.kt:77) 2021-07-29T16:43:24.375Z BROKEN plugin-btcli4j-gen.sh: \tat io.vincenzopalazzo.btcli4j.Plugin.getRawBlockByHeight(Plugin.kt:106) 2021-07-29T16:43:24.382Z BROKEN plugin-btcli4j-gen.sh: \tat jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) 2021-07-29T16:43:24.386Z BROKEN plugin-btcli4j-gen.sh: \tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2021-07-29T16:43:24.390Z BROKEN plugin-btcli4j-gen.sh: \tat java.base/java.lang.reflect.Method.invoke(Method.java:566) 2021-07-29T16:43:24.393Z BROKEN plugin-btcli4j-gen.sh: \tat jrpc.clightning.plugins.rpcmethods.RPCMethodReflection.doRun(RPCMethodReflection.java:36) 2021-07-29T16:43:24.397Z BROKEN plugin-btcli4j-gen.sh: \tat jrpc.clightning.plugins.CLightningPlugin.doMethods(CLightningPlugin.java:290) 2021-07-29T16:43:24.400Z BROKEN plugin-btcli4j-gen.sh: \tat jrpc.clightning.plugins.CLightningPlugin.start(CLightningPlugin.java:128) 2021-07-29T16:43:24.404Z BROKEN plugin-btcli4j-gen.sh: \tat io.vincenzopalazzo.btcli4j.AppKt.main(App.kt:26) 2021-07-29T16:43:24.407Z BROKEN plugin-btcli4j-gen.sh: \tat io.vincenzopalazzo.btcli4j.AppKt.main(App.kt) 2021-07-29T16:43:24.410Z BROKEN plugin-btcli4j-gen.sh: Caused by: java.io.EOFException: \n not found: limit=0 content=? 2021-07-29T16:43:24.414Z BROKEN plugin-btcli4j-gen.sh: \tat okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:332) 2021-07-29T16:43:24.417Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29) 2021-07-29T16:43:24.421Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:178) 2021-07-29T16:43:24.424Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106) 2021-07-29T16:43:24.428Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79) 2021-07-29T16:43:24.431Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2021-07-29T16:43:24.434Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) 2021-07-29T16:43:24.438Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2021-07-29T16:43:24.441Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) 2021-07-29T16:43:24.444Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2021-07-29T16:43:24.448Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) 2021-07-29T16:43:24.451Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2021-07-29T16:43:24.455Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) 2021-07-29T16:43:24.458Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2021-07-29T16:43:24.461Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) 2021-07-29T16:43:24.465Z BROKEN plugin-btcli4j-gen.sh: \tat okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) 2021-07-29T16:43:24.468Z BROKEN plugin-btcli4j-gen.sh: \tat io.github.clightning4j.litebtc.utils.okhttp.HttpFactory.makeRequest(HttpFactory.java:69) 2021-07-29T16:43:24.471Z BROKEN plugin-btcli4j-gen.sh: \tat io.github.clightning4j.litebtc.LiteBitcoinRPC.makeBitcoinRequest(LiteBitcoinRPC.java:63) 2021-07-29T16:43:24.475Z BROKEN plugin-btcli4j-gen.sh: \t... 11 more

Any idea what it is?

EDIT: I also noticed that the plugin is querying esplora for fee-estimates in pruned mode:

2021-07-29T16:53:39.600Z DEBUG plugin-btcli4j-gen.sh: Request to http://explorerzydxu5ecjrkwceayqybizmpjjznk5izmitf2modhcusuqlid.onion/api/fee-estimates with the following result [size=304 text={\"4\":2.455,\"20\":2.029,\"21\":2.029,\"1\":3.131,\"14\":2.029,\"13\":2.128?] 2021-07-29T16:53:39.605Z DEBUG plugin-btcli4j-gen.sh: EstimateFeeCommand: Estimate fee {\"4\":2.455,\"20\":2.029,\"21\":2.029,\"1\":3.131,\"14\":2.029,\"13\":2.128,\"19\":2.029,\"22\":2.029,\"15\":2.029,\"18\":2.029,\"23\":2.029,\"24\":2.029,\"25\":2.029,\"504\":2.029,\"17\":2.029,\"1008\":2.029,\"144\":2.029,\"7\":2.246,\"9\":2.246,\"16\":2.029,\"8\":2.246,\"5\":2.246,\"2\":3.131,\"11\":2.128,\"3\":3.131,\"12\":2.128,\"10\":2.128,\"6\":2.246}

Why is that? Why not querying the Bitcoin node it self?

vincenzopalazzo commented 3 years ago

Hi @sha-265,

Thanks so much to get this effort to catch this problem. Your idea about the plugin is perfectly fine.

Any idea what it is?

We can try to catch this error, so the core of the error is here io.github.clightning4j.litebtc.exceptions.LiteBitcoinRPCException: java.io.EOFException: \n not found: limit=0 content=?.

This contains useful information, like the exception is made by the bitcoind interface io.github.clightning4j.litebtc.exceptions.LiteBitcoinRPCException and the exception is from the answer get from Bitcoind, and the library to make the HTTP request throws this bad error.

A description of the error can be in the following issue and in particular, I want to focus the attention on this part

The EOFException is likely thrown when a pooled connection was re-used but has timed-out on the server in the meantime.

I'm wondering if your bitcoin core takes time to answer, this usually depends from the disk that you are using or the network. before fall into the debugging of this error (that I don't see on my server). Can you increase the timeout set on your bitcoin.conf file to rpcclienttimeout = 10000. this is the propriety that I have in my local machine and I use an old external hdd via USB 3.0.

Why is that? Why not querying the Bitcoin node itself?

The plugin is designed to be failure resistant, there are no reasons to use a backend plugin if your node will crash at the first error. For this reason, if your node has a delay in the answer, or your connection will go down for a couple of minutes the plugin try to avoid crash with a couple of operations.

This meas that to avoid the crash of the plugin makes the following phases:

  1. Talk with bitcoin core, if receive an error unexpected do 1a, or if the error is expected run 1b, or go one with success; 1a. Share the message to esplora API, if there is any error repeat the request 4 times with an increasing waiting time. 1b. Report the error to c-lightning, it know how handler it.

In your case, you are getting an error from bitcoin core and this error is unexpected, this means that the plugin will continue with esplora to complete with success the run of the command.

PS. Each method of the plugin, has this phase, this means that if an estimation fee will complete with esplora, the next command will use again bitcoin core, and repeat the phases described before.

I hope to receive your feedback about this new error. We did a good step to fix the autentication problem already. We will fix also this to run safally your c-lightning node.

Thanks again for your efforts.

sha-265 commented 3 years ago

Hi @vincenzopalazzo,

Sorry for the super long delay.

I'm wondering if your bitcoin core takes time to answer, this usually depends from the disk that you are using or the network. before fall into the debugging of this error (that I don't see on my server). Can you increase the timeout set on your bitcoin.conf file to rpcclienttimeout = 10000. this is the propriety that I have in my local machine and I use an old external hdd via USB 3.0.

I set rpcclienttimeout to 10000 but I still get the io.github.clightning4j.litebtc.exceptions.LiteBitcoinRPCException: java.io.EOFException: \n not found: limit=0 content=? error.

In your case, you are getting an error from bitcoin core and this error is unexpected, this means that the plugin will continue with esplora to complete with success the run of the command.

When running the fee-estimate command manually on my local full node, it's working fine. I see no reason it should be failover to Esplora.

Also, now I see after sometime the plugin is start spamming the log with the message:

DEBUG plugin-btcli4j-gen.sh: GetRawBlockByHeightBtc: terminate bitcoin core with error: Block not available (pruned data)

Any idea why it's spamming? Something like 100 messages per second.

Thanks.

EDIT: now I see some times it is taking the fee estimation from bitcoin core:

DEBUG plugin-btcli4j-gen.sh: EstimateFeeBtc: Estimate fee calculation from bitcoin core: 301

vincenzopalazzo commented 3 years ago

Hi @sha-265,

Sorry for the super long delay.

No delay here :)

I set rpcclienttimeout to 10000 but I still get the io.github.clightning4j.litebtc.exceptions.LiteBitcoinRPCException: java.io.EOFException: \n not found: limit=0 content=? error.

I'm not able to catch this error at this time, and I'm not able to reproduce it here, but here we have good news, that I will comment here below

DEBUG plugin-btcli4j-gen.sh: GetRawBlockByHeightBtc: terminate bitcoin core with error: Block not available (pruned data)

Any idea why it's spamming? Something like 100 messages per second.

The message GetRawBlockByHeightBtc: terminate bitcoin core with error: Block not available (pruned data) tells us that the plugin is working with the pruning mode available and it receives an error from Bitcoin core. This error means that c-lightning is asking for an old block and it is not present on the pruning mode. The main motivation here is that c-lightning asking for an old block is to verify the channels.

In addition, if you receive this message, I think your node is working in pruning mode, and the exception that you tell before it is an old exception.

Have this message spamming the log it is very bad, but it is the only way to debug the plugin for the moment. I'm in dev mode and I need to trace the life cycle of the plugin. However, I scheduled to remove some logs and add a new plugin command like backend info to get the actual mode of the plugin.

Can you try to delete the log file of c-lightning and restart c-lightning? I think you will have your node up and running with pruning mode turned on, and to check this you can see the message GetRawBlockByHeightBtc: terminate bitcoin core with error: Block not available (pruned data) spammed in the log.

I give you the correct details and answer your question?

Waiting for your feedback.

sha-265 commented 3 years ago

The message GetRawBlockByHeightBtc: terminate bitcoin core with error: Block not available (pruned data) tells us that the plugin is working with the pruning mode available and it receives an error from Bitcoin core. This error means that c-lightning is asking for an old block and it is not present on the pruning mode. The main motivation here is that c-lightning asking for an old block is to verify the channels.

That makes sense, but this message is repeating for each missing block more than once? Because it seems that a lot of blocks are missing, and that is something that didn't happen before.

Can you try to delete the log file of c-lightning and restart c-lightning? I think you will have your node up and running with pruning mode turned on, and to check this you can see the message GetRawBlockByHeightBtc: terminate bitcoin core with error: Block not available (pruned data) spammed in the log.

Yes I can see this message from time to time, but I wouldn't say it spammed the log, like the GetRawBlockByHeightBtc command.

Thanks @vincenzopalazzo

vincenzopalazzo commented 3 years ago

Hi @sha-265,

That makes sense, but this message is repeating for each missing block more than once? Because it seems that a lot of blocks are missing, and that is something that didn't happen before.

This message will shows for each call that c-lightning made for a block, maybe you could not see the message because it was on your local blockchain? and after a week, it is pruned?

There are two types of error that the plugin check if it is a bitcoin core error related to missing block or somethings like that, the plugin return the error to c-lightning, otherwise, if the bitcoin core has some connection error like in some your message before the plugin will try to address the issue of connection with esplora. If you receive some crash this means that the plugin tries to make the best that it can but just fails (and in this case your c-lightning node will crash)

Yes I can see this message from time to time, but I wouldn't say it spammed the log, like the GetRawBlockByHeightBtc command.

So, I think we address this issue, and sorry if the plugin is spamming your log, I will fix this in the next few days and give a plugin that doesn't spam nothings

P.S: If we fixed your issue, I will left this issue open and I will close in a couple of days, when you have the possibility to test it well on the system, and I also have this issue https://github.com/clightning4j/btcli4j/issues/45 to address your spamming problem.

Waiting for your feedback.

sha-265 commented 3 years ago

So, I think we address this issue, and sorry if the plugin is spamming your log, I will fix this in the next few days and give a plugin that doesn't spam nothings

Maybe I worded it wrong, but it's not so bad it's only when the log level set to debug level. But it can be annoying if you are trying to debug something else.

P.S: If we fixed your issue, I will left this issue open and I will close in a couple of days, when you have the possibility to test it well on the system, and I also have this issue #45 to address your spamming problem.

I hope we fixed it. I will try to look for other errors in the up coming days.

Thanks for the help!

vincenzopalazzo commented 3 years ago

Maybe I worded it wrong, but it's not so bad it's only when the log level set to debug level. But it can be annoying if you are trying to debug something else.

Yes, I was debugging this trick to manage the bitcoin error, but now it look good I think and maybe I can remove it and make the debug phase more lite.

I hope we fixed it. I will try to look for other errors in the up coming days.

Fantastic. Thanks for your efforts

vincenzopalazzo commented 3 years ago

Closing the issue and I considered this problem fixed.

Thanks