search

btcpayserver / btcpayserver-docker

Docker resources for hosting BTCPayServer easily
MIT License
590 stars 363 forks source link

lbtc issues with nbxplorer: unhandled exception in the indexer #912

Closed warioishere closed 2 months ago

warioishere commented 3 months ago

After an system update of my server running ubuntu server 22.04.4, nbxplorer doesnt sync with l-btc chain anymore. I restartet several times. I didnt do any changes to nbxplorer. I run newest version 1.13.5 The system is fully up to date.

Here are the logs from nbxplorer:

fail: NBXplorer.Indexer.LBTC: Unhandled exception in the indexer, retrying in 60 seconds
System.Net.Sockets.SocketException (111): Connection refused
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)

elementsd doesnt start up because of nbxplorer not running on l-btc:

************************
EXCEPTION: St19bad_optional_access       
bad optional access       
bitcoin in AppInit()       

2024-07-21T19:10:29Z Shutdown: In progress...

************************

elementsd is in loop not starting up.

Any hints would be nice as my peerswap daemon isnt running because of elementsd failing.

Thanks, let me know if you need any infos.

pavlenex commented 3 months ago

@Kukks can you check this one when you get a chance?

021-projects commented 3 months ago

+1

warioishere commented 3 months ago

Pushing the topic again, my peerswap and lbtc capabilities are down since two weeks now. Thats not so cool, I know @Kukks you have a lot to do, but I would really appreciate to get a hint or if you could take a look on that. I use peerswap quiet a lot and I need elements up again for that.

DariusIII commented 3 months ago

Same issue on my server too :(

fail: NBXplorer.Indexer.LBTC: Unhandled exception in the indexer, retrying in 60 seconds
System.Net.Sockets.SocketException (111): Connection refused
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
   at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
   at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 303
   at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 150
   at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 99
warioishere commented 2 months ago

dear @Kukks could you enlighten us about a fix for this bug? We were waiting now for more then a month, and I told you its kinda urgent for me. I cant do any l-btc trades anymore which I use to swap liq on my lightning node.

Its propably not so time consuming to release a hotfix. I just find it somehow weird that there is absolutly no response from the team about that. Dont think we are the only two suffering from this problem. I know you are not a official support dude but it also has something do do with respect to answer questions or maybe elaborate more on the timeline when this is going to be fixed.

warioishere commented 2 months ago

I also respect that you are busy, when at least give a little feedback on that too would be nice

Kukks commented 2 months ago

Please stop pushing me. I cannot enlighten you on this. I have too much on my plate as it is and I am probably the only person on the team that can be considered a liquid fan. This integration hasn't had that much traction and the maintenance burden around liquid is massive. If I ever have some free time and dedicated funding for this I'll look into it.

On Tue, Aug 27, 2024, 10:40 AM warioishere @.***> wrote:

dear @Kukks https://github.com/Kukks could you enlighten us about a fix for this bug? We were waiting now for more then a month, and I told you its kinda urgent for me. I cant do any l-btc trades anymore which I use to swap liq on my lightning node.

Its propably not so time consuming to release a hotfix. I just find it somehow weird that there is absolutly no response from the team about that. Dont think we are the only two suffering from this problem. I know you are not a official support dude but it also has something do do with respect to answer questions or maybe elaborate more on the timeline when this is going to be fixed.

— Reply to this email directly, view it on GitHub https://github.com/btcpayserver/btcpayserver-docker/issues/912#issuecomment-2311790051, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN357Q6EHTZU3AOM3X7UMLZTQUO3AVCNFSM6AAAAABLHC7QS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRG44TAMBVGE . You are receiving this because you were mentioned.Message ID: @.***>

warioishere commented 2 months ago

Please stop pushing me. I cannot enlighten you on this. I have too much on my plate as it is and I am probably the only person on the team that can be considered a liquid fan. This integration hasn't had that much traction and the maintenance burden around liquid is massive. If I ever have some free time and dedicated funding for this I'll look into it. … On Tue, Aug 27, 2024, 10:40 AM warioishere @.> wrote: dear @Kukks https://github.com/Kukks could you enlighten us about a fix for this bug? We were waiting now for more then a month, and I told you its kinda urgent for me. I cant do any l-btc trades anymore which I use to swap liq on my lightning node. Its propably not so time consuming to release a hotfix. I just find it somehow weird that there is absolutly no response from the team about that. Dont think we are the only two suffering from this problem. I know you are not a official support dude but it also has something do do with respect to answer questions or maybe elaborate more on the timeline when this is going to be fixed. — Reply to this email directly, view it on GitHub <#912 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN357Q6EHTZU3AOM3X7UMLZTQUO3AVCNFSM6AAAAABLHC7QS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRG44TAMBVGE . You are receiving this because you were mentioned.Message ID: @.>

I am not pushing, I am just asking because I dont get a reply from you, waiting for a month is not pushing. This means it could take another month to be fixed? Will have to move my main node then on another Node, thanks for letting me know

ndeet commented 2 months ago

The solution is to comment trim-headers in this file and do a . ./btcpay-setup.sh -i again

Seems the fix was merged in latest release 23.2.3 of a few hours ago (or rather seems it was there with 23.2.2 already).

@Kukks doing a new image (and update liquid.yml) with latest elementsd release should resolve this issue.

warioishere commented 2 months ago

big thanks to @ndeet for this investigation. I didnt really check elementsd for this bug, I just looked for solutions arround nbexplorer because I thought elementsd doesnt start up because of that. And the time this bug came up, the only thing that was updated, was nbxplorer and not elementsd, it stayed on the same version. So I really just focused solving the problem with nbx...

Do I understand it right, using trim-headers in the release of 23.2.3 is possible again? I really dont like the situation without trim-headers as it completly fills up the RAM after a while and I dont have endless resources. You know the problem :)

Kukks commented 2 months ago

big thanks to @ndeet for this investigation. I didnt really check elementsd for this bug, I just looked for solutions arround nbexplorer because I thought elementsd doesnt start up because of that. And the time this bug came up, the only thing that was updated, was nbxplorer and not elementsd, it stayed on the same version. So I really just focused solving the problem with nbx...

Do I understand it right, using trim-headers in the release of 23.2.3 is possible again? I really dont like the situation without trim-headers as it completly fills up the RAM after a while and I dont have endless resources. You know the problem :)

I have updated to 23.2.3 and pushed to this docker repo. Can you try now? I wonder if it will require a re-sync...

warioishere commented 2 months ago

big thanks to @ndeet for this investigation. I didnt really check elementsd for this bug, I just looked for solutions arround nbexplorer because I thought elementsd doesnt start up because of that. And the time this bug came up, the only thing that was updated, was nbxplorer and not elementsd, it stayed on the same version. So I really just focused solving the problem with nbx... Do I understand it right, using trim-headers in the release of 23.2.3 is possible again? I really dont like the situation without trim-headers as it completly fills up the RAM after a while and I dont have endless resources. You know the problem :)

I have updated to 23.2.3 and pushed to this docker repo. Can you try now? I wonder if it will require a re-sync...

thanks bud! If we stay within trimmed headers you actually dont need to rsync, if you uncomment trim-headers, a resync is required. As trim-headers is still used, a resync shouldnt be requiered.

Will test this out and let you know,

warioishere commented 2 months ago

@no resync required working again and elementsd starting up normally again :) :)

thanks to @ndeet and @Kukks for your time and your work, greatly appreciated!

ndeet commented 2 months ago

@Kukks @warioishere upgraded and re-enabled trim_headers, no re-index or re-sync needed it seems :rocket:

warioishere commented 2 months ago

now I am just waiting for lnd 0.18.3 to be merged and I am super happy :D :D I know I am a little pusher,

https://github.com/btcpayserver/btcpayserver-docker/pull/911 (See my post)

I wish I could help and I really already thought multiple times about helping you guys keeping the software like lnd bitcoincoire elements and etc up to date, but I am not familiar with github, testing and merging. Maybe in future i can help with that, but I need to get familiar with your testing methods and how everything works testing and putting things together.