search

btcpayserver / btcpayserver-docker

Docker resources for hosting BTCPayServer easily
MIT License
590 stars 364 forks source link

fail: NBXplorer.Indexer.DASH: Unhandled exception in the indexer, retrying / The node is starting... #944

Closed noahchrde closed 1 month ago

noahchrde commented 1 month ago

Here some logs for my error:

DASHD: 2024-10-24T16:06:00Z ProcessNewBlock : ACCEPTED 2024-10-24T16:06:00Z CMasternodeSync::ProcessTick -- disconnecting from recently synced peer=0 2024-10-24T16:06:00Z CMasternodeSync::ProcessTick -- disconnecting from recently synced peer=1 2024-10-24T16:06:00Z ThreadSocketHandler -- removing node: peer=1 nRefCount=1 fInbound=0 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:06:00Z ThreadRPCServer incorrect password attempt from 172.18.0.11:46794 2024-10-24T16:06:00Z New outbound peer connected: version: 70233, blocks=2160583, peer=3 (full-relay) 2024-10-24T16:06:00Z ThreadSocketHandler -- removing node: peer=0 nRefCount=1 fInbound=0 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:06:01Z New outbound peer connected: version: 70233, blocks=2160583, peer=5 (full-relay) 2024-10-24T16:06:01Z ThreadSocketHandler -- removing node: peer=4 nRefCount=1 fInbound=1 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:06:01Z New outbound peer connected: version: 70233, blocks=2160583, peer=6 (full-relay) 2024-10-24T16:06:03Z New outbound peer connected: version: 70233, blocks=2160582, peer=7 (full-relay) 2024-10-24T16:06:04Z [ProcessBlock] h[2160584] numCommitmentsRequired[0] numCommitmentsInNewBlock[0] 2024-10-24T16:06:04Z UpdateTip: new best=000000000000002341573fcc6646271b2a75aa299692973697bc299a271bd089 height=2160584 version=0x20000000 log2_work=79.296988 tx=54834438 date='2024-10-24T16:05:49Z' progress=1.000000 cache=0.5MiB(3684txo) evodb_cache=0.0MiB 2024-10-24T16:06:04Z ProcessNewBlock : ACCEPTED 2024-10-24T16:06:12Z CMasternodeSync::SwitchToNextAsset -- Completed MASTERNODE_SYNC_BLOCKCHAIN in 43s 2024-10-24T16:06:12Z CMasternodeSync::SwitchToNextAsset -- Starting MASTERNODE_SYNC_GOVERNANCE 2024-10-24T16:06:13Z New outbound peer connected: version: 70233, blocks=2160584, peer=8 (full-relay) 2024-10-24T16:06:14Z New outbound peer connected: version: 70233, blocks=2160584, peer=9 (full-relay) 2024-10-24T16:06:15Z New outbound peer connected: version: 70233, blocks=2160584, peer=10 (full-relay) 2024-10-24T16:06:16Z New outbound peer connected: version: 70233, blocks=2160584, peer=11 (block-relay) 2024-10-24T16:06:21Z ThreadSocketHandler -- removing node: peer=12 nRefCount=2 fInbound=1 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:06:33Z New outbound peer connected: version: 70233, blocks=2160584, peer=13 (block-relay) 2024-10-24T16:06:41Z [ProcessBlock] h[2160585] numCommitmentsRequired[0] numCommitmentsInNewBlock[0] 2024-10-24T16:06:41Z UpdateTip: new best=000000000000000557693cfb9131616945cad243432096f3383db7484b695de2 height=2160585 version=0x20000000 log2_work=79.296989 tx=54834444 date='2024-10-24T16:06:31Z' progress=1.000000 cache=0.5MiB(3716txo) evodb_cache=0.0MiB 2024-10-24T16:06:41Z ProcessNewBlock : ACCEPTED 2024-10-24T16:06:47Z ThreadSocketHandler -- removing node: peer=14 nRefCount=2 fInbound=1 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:06:48Z CMasternodeSync::SwitchToNextAsset -- Completed MASTERNODE_SYNC_GOVERNANCE in 36s 2024-10-24T16:06:48Z CMasternodeSync::SwitchToNextAsset -- Sync has finished 2024-10-24T16:07:17Z socket send error Broken pipe (32) (peer=15) 2024-10-24T16:07:17Z ThreadSocketHandler -- removing node: peer=15 nRefCount=1 fInbound=1 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:07:51Z [ProcessBlock] h[2160586] numCommitmentsRequired[0] numCommitmentsInNewBlock[0] 2024-10-24T16:07:51Z UpdateTip: new best=000000000000000a1b17605ecd33617451f876b3d736823c1916caf5ce4d81e8 height=2160586 version=0x20000000 log2_work=79.296990 tx=54834464 date='2024-10-24T16:07:36Z' progress=1.000000 cache=0.5MiB(3797txo) evodb_cache=0.0MiB 2024-10-24T16:07:51Z ProcessNewBlock : ACCEPTED 2024-10-24T16:07:52Z ThreadSocketHandler -- removing node: peer=16 nRefCount=2 fInbound=1 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:08:33Z ThreadSocketHandler -- removing node: peer=17 nRefCount=2 fInbound=1 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:09:18Z socket send error Broken pipe (32) (peer=18) 2024-10-24T16:09:18Z ThreadSocketHandler -- removing node: peer=18 nRefCount=1 fInbound=1 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:10:08Z ThreadSocketHandler -- removing node: peer=19 nRefCount=2 fInbound=1 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:10:13Z [ProcessBlock] h[2160587] numCommitmentsRequired[0] numCommitmentsInNewBlock[0] 2024-10-24T16:10:14Z UpdateTip: new best=000000000000001332edaac1f8d2fb77598d0967afeb725f33836dc274edcdbb height=2160587 version=0x20000000 log2_work=79.296991 tx=54834487 date='2024-10-24T16:09:55Z' progress=1.000000 cache=0.5MiB(3889txo) evodb_cache=0.0MiB 2024-10-24T16:10:14Z ProcessNewBlock : ACCEPTED 2024-10-24T16:10:45Z Loading addresses from DNS seed dnsseed.dash.org 2024-10-24T16:10:46Z 0 addresses found from DNS seeds 2024-10-24T16:10:46Z dnsseed thread exit 2024-10-24T16:10:46Z New outbound peer connected: version: 70233, blocks=2160587, peer=20 (full-relay) 2024-10-24T16:10:47Z ThreadSocketHandler -- removing node: peer=20 nRefCount=1 fInbound=0 m_masternode_connection=0 m_masternode_iqr_connection=0 2024-10-24T16:11:03Z ThreadSocketHandler -- removing node: peer=21 nRefCount=2 fInbound=1 m_masternode_connection=0 m_masternode_iqr_connection=0

nbxplorer: info: NBXplorer.Indexer.DASH: RPC connection successful info: NBXplorer.Indexer.DASH: Full node version detected: 200100 info: NBXplorer.Indexer.DASH: Has txindex support info: NBXplorer.Indexer.DASH: NBXplorer is correctly whitelisted by the node info: NBXplorer.Events: DASH: Node state changed: NotStarted => NBXplorerSynching info: NBXplorer.Indexer.DASH: Node disconnected (Unexpected exception while connecting to socket) info: NBXplorer.Events: DASH: Node state changed: NBXplorerSynching => NotStarted fail: NBXplorer.Indexer.DASH: Unhandled exception in the indexer, retrying in 50 seconds System.OperationCanceledException: The peer has been disconnected at NBXplorer.Backend.Indexer.AskNextHeaders(Node node, CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 382 at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 215 at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 220 at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 148 at NBXplorer.Backend.Indexer.IndexerLoop() in /source/NBXplorer/Backend/Indexer.cs:line 95 info: NBXplorer.Events: DOGE: New block 924d03f32966b3e972087544f2d367524522ce4153bb27b27d8155064c087103 (5432973) info: NBXplorer.Events: LTC: New block afec9b1acb9c302e5cc0d27844e050e6ead0efc212758a605e5ed184372050cc (2779321) info: NBXplorer.Indexer.DASH: TCP Connection succeed, handshaking... info: NBXplorer.Indexer.DASH: Handshaked info: NBXplorer.Indexer.DASH: Testing RPC connection to http://dashd:9998/ info: NBXplorer.Indexer.DASH: RPC connection successful info: NBXplorer.Indexer.DASH: Full node version detected: 200100 info: NBXplorer.Indexer.DASH: Has txindex support info: NBXplorer.Indexer.DASH: NBXplorer is correctly whitelisted by the node info: NBXplorer.Events: DASH: Node state changed: NotStarted => NBXplorerSynching info: NBXplorer.Indexer.DASH: Node disconnected (Unexpected exception while connecting to socket) info: NBXplorer.Events: DASH: Node state changed: NBXplorerSynching => NotStarted fail: NBXplorer.Indexer.DASH: Unhandled exception in the indexer, retrying in 55 seconds System.OperationCanceledException: The peer has been disconnected at NBXplorer.Backend.Indexer.AskNextHeaders(Node node, CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 382 at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 215 at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 220 at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 148 at NBXplorer.Backend.Indexer.IndexerLoop() in /source/NBXplorer/Backend/Indexer.cs:line 95 info: NBXplorer.Events: DOGE: New block 73af13c5b5f7df154ef186508ddf1e639fa258dd216aa0a13b305ed984b4f752 (5432974) info: NBXplorer.Events: LTC: New block b1bb26c37fdbd0e3072a90e6e682c107c994797b55262b373877438d6ce665a9 (2779322) info: NBXplorer.Indexer.DASH: TCP Connection succeed, handshaking... info: NBXplorer.Indexer.DASH: Handshaked info: NBXplorer.Indexer.DASH: Testing RPC connection to http://dashd:9998/ info: NBXplorer.Indexer.DASH: RPC connection successful info: NBXplorer.Indexer.DASH: Full node version detected: 200100 info: NBXplorer.Indexer.DASH: Has txindex support info: NBXplorer.Indexer.DASH: NBXplorer is correctly whitelisted by the node info: NBXplorer.Events: DASH: Node state changed: NotStarted => NBXplorerSynching info: NBXplorer.Indexer.DASH: Node disconnected (Unexpected exception while connecting to socket) info: NBXplorer.Events: DASH: Node state changed: NBXplorerSynching => NotStarted fail: NBXplorer.Indexer.DASH: Unhandled exception in the indexer, retrying in 60 seconds System.OperationCanceledException: The peer has been disconnected at NBXplorer.Backend.Indexer.AskNextHeaders(Node node, CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 382 at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 215 at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 220 at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 148 at NBXplorer.Backend.Indexer.IndexerLoop() in /source/NBXplorer/Backend/Indexer.cs:line 95

I tried to restart everything and restarted hole server, still not working.

Every other coin is still working fine. Also DASH was working for months and stopped working without any changes from my side: nlvf9divjkndmcl

Thanks for your help.

huggix commented 1 month ago

I'm experiencing the same issue. Our btcpayserver is new, so I thought it was a setup thing and have been debugging it for days.

LTC, DOGE, BTC, XMR are working and running as expected.

From the Dash container logs, I can see the node is fully synched and participating in the network.

My container logs for generated_nbxplorer_1 (that keeps repeating):

info: NBXplorer.Indexer.DASH: TCP Connection succeed, handshaking...
info: NBXplorer.Indexer.DASH: Handshaked
info: NBXplorer.Indexer.DASH: Testing RPC connection to http://dashd:9998/
info: NBXplorer.Indexer.DASH: RPC connection successful
info: NBXplorer.Indexer.DASH: Full node version detected: 200100
info: NBXplorer.Indexer.DASH: Has txindex support
info: NBXplorer.Indexer.DASH: NBXplorer is correctly whitelisted by the node
info: NBXplorer.Events: DASH: Node state changed: NotStarted => NBXplorerSynching
info: NBXplorer.Indexer.DASH: Node disconnected (Unexpected exception while connecting to socket)
info: NBXplorer.Events: DASH: Node state changed: NBXplorerSynching => NotStarted
fail: NBXplorer.Indexer.DASH: Unhandled exception in the indexer, retrying in 40 seconds
System.OperationCanceledException: The peer has been disconnected
   at NBXplorer.Backend.Indexer.AskNextHeaders(Node node, CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 383
   at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 216
   at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 221
   at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 149
   at NBXplorer.Backend.Indexer.IndexerLoop() in /source/NBXplorer/Backend/Indexer.cs:line 96
noahchrde commented 1 month ago

I'm experiencing the same issue. Our btcpayserver is new, so I thought it was a setup thing and have been debugging it for days.

LTC, DOGE, BTC, XMR are working and running as expected.

From the Dash container logs, I can see the node is fully synched and participating in the network.

My container logs for generated_nbxplorer_1 (that keeps repeating):

info: NBXplorer.Indexer.DASH: TCP Connection succeed, handshaking...
info: NBXplorer.Indexer.DASH: Handshaked
info: NBXplorer.Indexer.DASH: Testing RPC connection to http://dashd:9998/
info: NBXplorer.Indexer.DASH: RPC connection successful
info: NBXplorer.Indexer.DASH: Full node version detected: 200100
info: NBXplorer.Indexer.DASH: Has txindex support
info: NBXplorer.Indexer.DASH: NBXplorer is correctly whitelisted by the node
info: NBXplorer.Events: DASH: Node state changed: NotStarted => NBXplorerSynching
info: NBXplorer.Indexer.DASH: Node disconnected (Unexpected exception while connecting to socket)
info: NBXplorer.Events: DASH: Node state changed: NBXplorerSynching => NotStarted
fail: NBXplorer.Indexer.DASH: Unhandled exception in the indexer, retrying in 40 seconds
System.OperationCanceledException: The peer has been disconnected
   at NBXplorer.Backend.Indexer.AskNextHeaders(Node node, CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 383
   at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 216
   at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 221
   at NBXplorer.Backend.Indexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backend/Indexer.cs:line 149
   at NBXplorer.Backend.Indexer.IndexerLoop() in /source/NBXplorer/Backend/Indexer.cs:line 96

Hope it will be fixed soon. We used dash a lot. Maybe it happens with one of the last updates, idk.

Waiting for some knowledge here.

huggix commented 1 month ago

When did Dash stopped working for you?

The btcpayserver/dash image was updated 4 months ago

The last relevant change to the btcpayserver-docker repo i see is from june 14th, where btcpayserver-dash was updated from 20.0.2 to 20.1.0

noahchrde commented 1 month ago

Stopped working i think around 2 weeks ago or something. I didnt change anything. Only sometimes I pressed the update button. - Thats all. 😅

huggix commented 1 month ago

An issue has been created on the NBXplorer github as well and the author has responded: https://github.com/dgarage/NBXplorer/issues/481

Seems like a fix has been created and pushed: https://github.com/MetacoSA/NBitcoin/pull/1235

noahchrde commented 1 month ago

Oh interesting. Good to have that fixed. Found nothing on the net when created this issue here. Hope we will get working docker images soon.

NicolasDorier commented 1 month ago

Will update NBXplorer soon today.

NicolasDorier commented 1 month ago

Updated NBXplorer to 2.5.8, just update https://github.com/btcpayserver/btcpayserver-docker/commit/d7e603a3a4696eb52c846e19fa55a4797ffccec6

noahchrde commented 1 month ago

Thank you so much that works out. After update & restart everything is fixed. Thank you so much for fast work. 🙏

huggix commented 1 month ago

Working! Thank you for a quick fix Nicolas.

For other who might find this issue:

Update the NBXplorer version in: ~/btcpayserver-docker/docker-compose-generator/docker-fragments/nbxplorer.yml

Run Btcpay-setup -i