stuck at NBXplorer is synchronizing after updating

adspirit commented 2 years ago

Good day,

I was running on v1.4x and everything was working fine. Then updated to latest version, noticed that syncing headers height was working fine (NBXplorer headers height: 757366) but NBXplorer sync is stuck (NBXplorer is synchronizing... (Height: 691671)

I have prune setup "opt-save-storage-s" and below is my log:

2022-10-06T16:28:27.532744022Z info: Configuration:  Data Directory: /datadir
2022-10-06T16:28:27.532850740Z info: Configuration:  Configuration File: /datadir/Main/settings.config
2022-10-06T16:28:27.554119505Z info: Configuration:  Network: Mainnet
2022-10-06T16:28:27.561305316Z info: Configuration:  Supported chains: BTC
2022-10-06T16:28:27.561325402Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:28:27.643858711Z info: Configuration:  Network: Mainnet
2022-10-06T16:28:27.643974575Z info: Configuration:  Supported chains: BTC
2022-10-06T16:28:27.644011534Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:28:27.804561254Z info: NBXplorer.DatabaseSetup: Postgres services activated
2022-10-06T16:28:28.205858106Z info: Explorer:       Trimming the event table if needed...
2022-10-06T16:28:28.289841507Z warn: NBXplorer.PostgresMigration: A legacy DBTrie database has been previously migrated to postgres and is still present. You can safely delete it if you do not expect using it in the future. To delete the old DBTrie database, start NBXplorer with --deleteaftermigration (or environment variable: NBXPLORER_DELETEAFTERMIGRATION=1)
2022-10-06T16:28:28.433770352Z Hosting environment: Production
2022-10-06T16:28:28.433798952Z Content root path: /app/
2022-10-06T16:28:28.433836979Z Now listening on: http://0.0.0.0:32838
2022-10-06T16:28:28.433842033Z Application started. Press Ctrl+C to shut down.
2022-10-06T16:28:29.391614899Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 0 seconds
2022-10-06T16:28:29.391635064Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:28:29.391795812Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:28:29.391813400Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:28:29.391817386Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:28:29.391821332Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:28:29.391831494Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:28:29.391834870Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:28:29.391857206Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:28:29.391861250Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:28:29.394657669Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 5 seconds
2022-10-06T16:28:29.394674943Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:28:29.394678706Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:28:29.394682144Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:28:29.394695179Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:28:29.394699186Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:28:29.394702545Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:28:29.394750455Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:28:29.394780350Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:28:29.394786519Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:28:34.408093887Z info: NBXplorer.Indexer.BTC: TCP Connection succeed, handshaking...
2022-10-06T16:28:34.452657382Z info: NBXplorer.Indexer.BTC: Handshaked
2022-10-06T16:28:34.458762212Z info: NBXplorer.Indexer.BTC: Testing RPC connection to http://bitcoind:43782/
2022-10-06T16:28:34.848685995Z info: NBXplorer.Indexer.BTC: RPC connection successful
2022-10-06T16:28:35.105296887Z info: NBXplorer.Indexer.BTC: Full node version detected: 230000
2022-10-06T16:28:35.116341822Z info: NBXplorer.Indexer.BTC: NBXplorer is correctly whitelisted by the node
2022-10-06T16:28:35.235653328Z info: NBXplorer.Events: BTC: Node state changed: NotStarted => CoreSynching
2022-10-06T16:28:50.014384817Z Application is shutting down...
2022-10-06T16:28:57.836246482Z info: Configuration:  Data Directory: /datadir
2022-10-06T16:28:57.836325188Z info: Configuration:  Configuration File: /datadir/Main/settings.config
2022-10-06T16:28:57.857642989Z info: Configuration:  Network: Mainnet
2022-10-06T16:28:57.864938563Z info: Configuration:  Supported chains: BTC
2022-10-06T16:28:57.864965479Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:28:57.947887972Z info: Configuration:  Network: Mainnet
2022-10-06T16:28:57.947964811Z info: Configuration:  Supported chains: BTC
2022-10-06T16:28:57.947989737Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:28:58.098692744Z info: NBXplorer.DatabaseSetup: Postgres services activated
2022-10-06T16:28:58.521226359Z info: Explorer:       Trimming the event table if needed...
2022-10-06T16:28:58.622647709Z warn: NBXplorer.PostgresMigration: A legacy DBTrie database has been previously migrated to postgres and is still present. You can safely delete it if you do not expect using it in the future. To delete the old DBTrie database, start NBXplorer with --deleteaftermigration (or environment variable: NBXPLORER_DELETEAFTERMIGRATION=1)
2022-10-06T16:28:58.789426591Z Hosting environment: Production
2022-10-06T16:28:58.789459329Z Content root path: /app/
2022-10-06T16:28:58.789464085Z Now listening on: http://0.0.0.0:32838
2022-10-06T16:28:58.789467854Z Application started. Press Ctrl+C to shut down.
2022-10-06T16:29:00.748250644Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 0 seconds
2022-10-06T16:29:00.748285704Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:29:00.748290887Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:29:00.748294965Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:29:00.748298806Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:29:00.748303135Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:29:00.748307146Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:29:00.748322219Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:29:00.748325805Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:29:00.748329218Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:29:00.751037497Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 5 seconds
2022-10-06T16:29:00.751060579Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:29:00.751066329Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:29:00.751071423Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:29:00.751076434Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:29:00.751081907Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:29:00.751087321Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:29:00.751091558Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:29:00.751094897Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:29:00.751098240Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:29:05.768362466Z info: NBXplorer.Indexer.BTC: TCP Connection succeed, handshaking...
2022-10-06T16:29:05.808551848Z info: NBXplorer.Indexer.BTC: Handshaked
2022-10-06T16:29:05.814493289Z info: NBXplorer.Indexer.BTC: Testing RPC connection to http://bitcoind:43782/
2022-10-06T16:29:06.143119026Z info: NBXplorer.Indexer.BTC: RPC connection successful
2022-10-06T16:29:06.387589356Z info: NBXplorer.Indexer.BTC: Full node version detected: 230000
2022-10-06T16:29:06.398484609Z info: NBXplorer.Indexer.BTC: NBXplorer is correctly whitelisted by the node
2022-10-06T16:29:06.446010619Z info: NBXplorer.Events: BTC: Node state changed: NotStarted => CoreSynching
2022-10-06T16:29:19.354388877Z Application is shutting down...
2022-10-06T16:29:26.481843014Z info: Configuration:  Data Directory: /datadir
2022-10-06T16:29:26.481889421Z info: Configuration:  Configuration File: /datadir/Main/settings.config
2022-10-06T16:29:26.504664728Z info: Configuration:  Network: Mainnet
2022-10-06T16:29:26.509850350Z info: Configuration:  Supported chains: BTC
2022-10-06T16:29:26.509874796Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:29:26.594830965Z info: Configuration:  Network: Mainnet
2022-10-06T16:29:26.595171364Z info: Configuration:  Supported chains: BTC
2022-10-06T16:29:26.595192684Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:29:26.766584651Z info: NBXplorer.DatabaseSetup: Postgres services activated
2022-10-06T16:29:27.819286973Z info: Explorer:       Trimming the event table if needed...
2022-10-06T16:29:27.921742380Z warn: NBXplorer.PostgresMigration: A legacy DBTrie database has been previously migrated to postgres and is still present. You can safely delete it if you do not expect using it in the future. To delete the old DBTrie database, start NBXplorer with --deleteaftermigration (or environment variable: NBXPLORER_DELETEAFTERMIGRATION=1)
2022-10-06T16:29:28.052994795Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 0 seconds
2022-10-06T16:29:28.053028198Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:29:28.053033434Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:29:28.053037558Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:29:28.053041622Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:29:28.053045644Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:29:28.053049504Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:29:28.053053167Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:29:28.053069722Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:29:28.053073374Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:29:28.055710314Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 5 seconds
2022-10-06T16:29:28.055726480Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:29:28.055730634Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:29:28.055734178Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:29:28.055737487Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:29:28.055741054Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:29:28.055744399Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:29:28.055747655Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:29:28.055750979Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:29:28.055754231Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:29:28.146393758Z Hosting environment: Production
2022-10-06T16:29:28.146455281Z Content root path: /app/
2022-10-06T16:29:28.146459971Z Now listening on: http://0.0.0.0:32838
2022-10-06T16:29:28.146463804Z Application started. Press Ctrl+C to shut down.
2022-10-06T16:29:33.085124928Z info: NBXplorer.Indexer.BTC: TCP Connection succeed, handshaking...
2022-10-06T16:29:33.136668777Z info: NBXplorer.Indexer.BTC: Handshaked
2022-10-06T16:29:33.143328100Z info: NBXplorer.Indexer.BTC: Testing RPC connection to http://bitcoind:43782/
2022-10-06T16:29:33.285219529Z info: NBXplorer.Indexer.BTC: RPC connection successful
2022-10-06T16:29:33.667506862Z info: NBXplorer.Indexer.BTC: Full node version detected: 230000
2022-10-06T16:29:33.697579999Z info: NBXplorer.Indexer.BTC: NBXplorer is correctly whitelisted by the node
2022-10-06T16:29:34.034154853Z info: NBXplorer.Events: BTC: Node state changed: NotStarted => CoreSynching
docker logs -f --timestamps generated_nbxplorer_1^C
root@localhost:~/BTCPayServer/btcpayserver-docker# docker logs -f --timestamps generated_nbxplorer_1
2022-10-06T16:28:27.532744022Z info: Configuration:  Data Directory: /datadir
2022-10-06T16:28:27.532850740Z info: Configuration:  Configuration File: /datadir/Main/settings.config
2022-10-06T16:28:27.554119505Z info: Configuration:  Network: Mainnet
2022-10-06T16:28:27.561305316Z info: Configuration:  Supported chains: BTC
2022-10-06T16:28:27.561325402Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:28:27.643858711Z info: Configuration:  Network: Mainnet
2022-10-06T16:28:27.643974575Z info: Configuration:  Supported chains: BTC
2022-10-06T16:28:27.644011534Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:28:27.804561254Z info: NBXplorer.DatabaseSetup: Postgres services activated
2022-10-06T16:28:28.205858106Z info: Explorer:       Trimming the event table if needed...
2022-10-06T16:28:28.289841507Z warn: NBXplorer.PostgresMigration: A legacy DBTrie database has been previously migrated to postgres and is still present. You can safely delete it if you do not expect using it in the future. To delete the old DBTrie database, start NBXplorer with --deleteaftermigration (or environment variable: NBXPLORER_DELETEAFTERMIGRATION=1)
2022-10-06T16:28:28.433770352Z Hosting environment: Production
2022-10-06T16:28:28.433798952Z Content root path: /app/
2022-10-06T16:28:28.433836979Z Now listening on: http://0.0.0.0:32838
2022-10-06T16:28:28.433842033Z Application started. Press Ctrl+C to shut down.
2022-10-06T16:28:29.391614899Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 0 seconds
2022-10-06T16:28:29.391635064Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:28:29.391795812Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:28:29.391813400Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:28:29.391817386Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:28:29.391821332Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:28:29.391831494Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:28:29.391834870Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:28:29.391857206Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:28:29.391861250Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:28:29.394657669Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 5 seconds
2022-10-06T16:28:29.394674943Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:28:29.394678706Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:28:29.394682144Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:28:29.394695179Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:28:29.394699186Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:28:29.394702545Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:28:29.394750455Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:28:29.394780350Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:28:29.394786519Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:28:34.408093887Z info: NBXplorer.Indexer.BTC: TCP Connection succeed, handshaking...
2022-10-06T16:28:34.452657382Z info: NBXplorer.Indexer.BTC: Handshaked
2022-10-06T16:28:34.458762212Z info: NBXplorer.Indexer.BTC: Testing RPC connection to http://bitcoind:43782/
2022-10-06T16:28:34.848685995Z info: NBXplorer.Indexer.BTC: RPC connection successful
2022-10-06T16:28:35.105296887Z info: NBXplorer.Indexer.BTC: Full node version detected: 230000
2022-10-06T16:28:35.116341822Z info: NBXplorer.Indexer.BTC: NBXplorer is correctly whitelisted by the node
2022-10-06T16:28:35.235653328Z info: NBXplorer.Events: BTC: Node state changed: NotStarted => CoreSynching
2022-10-06T16:28:50.014384817Z Application is shutting down...
2022-10-06T16:28:57.836246482Z info: Configuration:  Data Directory: /datadir
2022-10-06T16:28:57.836325188Z info: Configuration:  Configuration File: /datadir/Main/settings.config
2022-10-06T16:28:57.857642989Z info: Configuration:  Network: Mainnet
2022-10-06T16:28:57.864938563Z info: Configuration:  Supported chains: BTC
2022-10-06T16:28:57.864965479Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:28:57.947887972Z info: Configuration:  Network: Mainnet
2022-10-06T16:28:57.947964811Z info: Configuration:  Supported chains: BTC
2022-10-06T16:28:57.947989737Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:28:58.098692744Z info: NBXplorer.DatabaseSetup: Postgres services activated
2022-10-06T16:28:58.521226359Z info: Explorer:       Trimming the event table if needed...
2022-10-06T16:28:58.622647709Z warn: NBXplorer.PostgresMigration: A legacy DBTrie database has been previously migrated to postgres and is still present. You can safely delete it if you do not expect using it in the future. To delete the old DBTrie database, start NBXplorer with --deleteaftermigration (or environment variable: NBXPLORER_DELETEAFTERMIGRATION=1)
2022-10-06T16:28:58.789426591Z Hosting environment: Production
2022-10-06T16:28:58.789459329Z Content root path: /app/
2022-10-06T16:28:58.789464085Z Now listening on: http://0.0.0.0:32838
2022-10-06T16:28:58.789467854Z Application started. Press Ctrl+C to shut down.
2022-10-06T16:29:00.748250644Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 0 seconds
2022-10-06T16:29:00.748285704Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:29:00.748290887Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:29:00.748294965Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:29:00.748298806Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:29:00.748303135Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:29:00.748307146Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:29:00.748322219Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:29:00.748325805Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:29:00.748329218Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:29:00.751037497Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 5 seconds
2022-10-06T16:29:00.751060579Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:29:00.751066329Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:29:00.751071423Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:29:00.751076434Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:29:00.751081907Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:29:00.751087321Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:29:00.751091558Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:29:00.751094897Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:29:00.751098240Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:29:05.768362466Z info: NBXplorer.Indexer.BTC: TCP Connection succeed, handshaking...
2022-10-06T16:29:05.808551848Z info: NBXplorer.Indexer.BTC: Handshaked
2022-10-06T16:29:05.814493289Z info: NBXplorer.Indexer.BTC: Testing RPC connection to http://bitcoind:43782/
2022-10-06T16:29:06.143119026Z info: NBXplorer.Indexer.BTC: RPC connection successful
2022-10-06T16:29:06.387589356Z info: NBXplorer.Indexer.BTC: Full node version detected: 230000
2022-10-06T16:29:06.398484609Z info: NBXplorer.Indexer.BTC: NBXplorer is correctly whitelisted by the node
2022-10-06T16:29:06.446010619Z info: NBXplorer.Events: BTC: Node state changed: NotStarted => CoreSynching
2022-10-06T16:29:19.354388877Z Application is shutting down...
2022-10-06T16:29:26.481843014Z info: Configuration:  Data Directory: /datadir
2022-10-06T16:29:26.481889421Z info: Configuration:  Configuration File: /datadir/Main/settings.config
2022-10-06T16:29:26.504664728Z info: Configuration:  Network: Mainnet
2022-10-06T16:29:26.509850350Z info: Configuration:  Supported chains: BTC
2022-10-06T16:29:26.509874796Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:29:26.594830965Z info: Configuration:  Network: Mainnet
2022-10-06T16:29:26.595171364Z info: Configuration:  Supported chains: BTC
2022-10-06T16:29:26.595192684Z info: Configuration:  DBCache: 50 MB
2022-10-06T16:29:26.766584651Z info: NBXplorer.DatabaseSetup: Postgres services activated
2022-10-06T16:29:27.819286973Z info: Explorer:       Trimming the event table if needed...
2022-10-06T16:29:27.921742380Z warn: NBXplorer.PostgresMigration: A legacy DBTrie database has been previously migrated to postgres and is still present. You can safely delete it if you do not expect using it in the future. To delete the old DBTrie database, start NBXplorer with --deleteaftermigration (or environment variable: NBXPLORER_DELETEAFTERMIGRATION=1)
2022-10-06T16:29:28.052994795Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 0 seconds
2022-10-06T16:29:28.053028198Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:29:28.053033434Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:29:28.053037558Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:29:28.053041622Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:29:28.053045644Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:29:28.053049504Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:29:28.053053167Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:29:28.053069722Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:29:28.053073374Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:29:28.055710314Z fail: NBXplorer.Indexer.BTC: Unhandled exception in the indexer, retrying in 5 seconds
2022-10-06T16:29:28.055726480Z System.Net.Sockets.SocketException (111): Connection refused
2022-10-06T16:29:28.055730634Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2022-10-06T16:29:28.055734178Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2022-10-06T16:29:28.055737487Z    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2022-10-06T16:29:28.055741054Z    at NBitcoin.Protocol.Connectors.DefaultEndpointConnector.ConnectSocket(Socket socket, EndPoint endpoint, NodeConnectionParameters nodeConnectionParameters, CancellationToken cancellationToken)
2022-10-06T16:29:28.055744399Z    at NBitcoin.Protocol.Node.ConnectAsync(Network network, EndPoint endpoint, NetworkAddress peer, NodeConnectionParameters parameters)
2022-10-06T16:29:28.055747655Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.ConnectNode(CancellationToken token, Boolean forceRestart) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 209
2022-10-06T16:29:28.055750979Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoopCore(CancellationToken token) in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 76
2022-10-06T16:29:28.055754231Z    at NBXplorer.Backends.Postgres.PostgresIndexers.PostgresIndexer.IndexerLoop() in /source/NBXplorer/Backends/Postgres/PostgresIndexers.cs:line 55
2022-10-06T16:29:28.146393758Z Hosting environment: Production
2022-10-06T16:29:28.146455281Z Content root path: /app/
2022-10-06T16:29:28.146459971Z Now listening on: http://0.0.0.0:32838
2022-10-06T16:29:28.146463804Z Application started. Press Ctrl+C to shut down.
2022-10-06T16:29:33.085124928Z info: NBXplorer.Indexer.BTC: TCP Connection succeed, handshaking...
2022-10-06T16:29:33.136668777Z info: NBXplorer.Indexer.BTC: Handshaked
2022-10-06T16:29:33.143328100Z info: NBXplorer.Indexer.BTC: Testing RPC connection to http://bitcoind:43782/
2022-10-06T16:29:33.285219529Z info: NBXplorer.Indexer.BTC: RPC connection successful
2022-10-06T16:29:33.667506862Z info: NBXplorer.Indexer.BTC: Full node version detected: 230000
2022-10-06T16:29:33.697579999Z info: NBXplorer.Indexer.BTC: NBXplorer is correctly whitelisted by the node
2022-10-06T16:29:34.034154853Z info: NBXplorer.Events: BTC: Node state changed: NotStarted => CoreSynching

tried rebooting server and followed this guide but with no luck.

Please advise.

Thanks and Regards,

NicolasDorier commented 2 years ago

The error seems to be your bitcoin node having troubles. Can you check bitcoind logs?

How much RAM you have?

adspirit commented 2 years ago

CPU:4 vCore RAM: 8GB SSD: 160 GB

In attempt to fix it I deleted the whole data dir and now it is validating blocks from start, currently at 75% will confirm the status soon

NicolasDorier commented 2 years ago

@adspirit for this SSD size, you might get issue with disk space, and that may have been the issue, are you using pruning?

If not,

BTCPAYGEN_ADDITIONAL_FRAGMENTS="opt-save-storage-s"
. btcpay-setup.sh -i

adspirit commented 2 years ago

Yes I use pruning (s). Before deleting dir I tried different pruning (xs, xxs), pruneheight changed but still was stuck.

currently it's at 86%, will report once it is stuck or completed.

NicolasDorier commented 2 years ago

86% seems to me high. If you used pruning it shouldn't go that high. -s for example, should be around 70 GB

ndeet commented 2 years ago

@NicolasDorier I think he means the sync progress is 86% and not the disk usage

adspirit commented 2 years ago

86% seems to me high. If you used pruning it shouldn't go that high. -s for example, should be around 70 GB

I was talking about validating blocks, it finished validating but again stuck at NBXplorer sync

BTC NBXplorer headers height: 757495 The node is synchronized (Height: 757495) NBXplorer is synchronizing... (Height: 691671)

{
  "chain": "main",
  "blocks": 757496,
  "headers": 757496,
  "bestblockhash": "00000000000000000002e08d27f5f3c8a1a7cf04386f64edadf2103605124811",
  "difficulty": 31360548173144.85,
  "time": 1665135775,
  "mediantime": 1665131899,
  "verificationprogress": 0.9999990955028843,
  "initialblockdownload": false,
  "chainwork": "00000000000000000000000000000000000000003697666fdd119afd697df678",
  "size_on_disk": 48763181486,
  "pruned": true,
  "pruneheight": 721296,
  "automatic_pruning": true,
  "prune_target_size": 52428800000,
  "warnings": ""
}

I'm not sure how to resolve this issue.

NicolasDorier commented 2 years ago

With so many block behind for NBXplorer, my guess of what is happening is that NBXplorer can't request blocks that far because those have been pruned.

In this case, you should remove pruning, but then you need enough disk space.

If you just want to recover your money, just start a new instance on a separate VPS like lunanode, create a store with your hardware wallet like you did before, then rescan the wallet. You will see the balance back.

btcpayserver / btcpayserver-docker

stuck at NBXplorer is synchronizing after updating #708