Exception when trying to rescan (a second time)

djpnewton commented 5 years ago

I am trying to trigger a rescan from the command line. I did it once but now it fails due to an exception.

I think it is because the 'Repository' constructor calls repo.SetIndexProgress(null); before some sort of db transaction is initialized

info: Configuration:  BTC: Rescanning the chain...
crit: Explorer:       BTC: Unhandled exception in the repository
System.NullReferenceException: Object reference not set to an instance of an object.
   at NBXplorer.Repository.<>c__DisplayClass19_0.<SetIndexProgress>b__0(Transaction tx) in ...\NBXplorer\NBXplorer\Repository.cs:line 348
   at NBXplorer.DBriizeTransactionContext.<>c__DisplayClass16_0.<DoAsyncCore>b__0() in ...\NBXplorer\NBXplorer\DBriizeTransactionContext.cs:line 92
   at NBXplorer.DBriizeTransactionContext.Loop() in ...\NBXplorer\NBXplorer\DBriizeTransactionContext.cs:line 53

NicolasDorier commented 5 years ago

checking this now

NicolasDorier commented 5 years ago

so this should never happen.

tx.RemoveKey($"{_Suffix}IndexProgress", ""); which indicate a condition where tx is null. I don't see how this can happen.

NicolasDorier commented 5 years ago

Trying to reproduce...

NicolasDorier commented 5 years ago

Trying --btcrescan=1 --btcstartheight=570000 or something like this, did not crashed.

NicolasDorier commented 5 years ago

Have you more info which could help me to repro?

djpnewton commented 5 years ago

Im using testnet.

I happens reliably. I start with --btcrescan=1 --btcstartheight=1400000

Here is my settings.config:

####Common Commands####
####If Bitcoin Core is running with default settings, you should not need to modify this file####
####All those options can be passed by through command like arguments (ie `-port=19382`)####
## This is the RPC Connection to your node
#btc.rpc.url=http://127.0.0.1:8332/
btc.rpc.url=http://10.50.1.100:18332/
#By user name and password
#btc.rpc.user=bitcoinuser
btc.rpc.user=user
#btc.rpc.password=bitcoinpassword
btc.rpc.password=pass
#By cookie file
#btc.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#btc.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#btc.node.endpoint=127.0.0.1:BTC
btc.node.endpoint=10.50.1.100

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#btc.startheight=-1
## rescan forces a rescan from startheight
#btc.rescan=0
## This is the RPC Connection to your node
#btx.rpc.url=http://127.0.0.1:8556/
#By user name and password
#btx.rpc.user=bitcoinuser
#btx.rpc.password=bitcoinpassword
#By cookie file
#btx.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#btx.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#btx.node.endpoint=127.0.0.1:BTX

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#btx.startheight=-1
## rescan forces a rescan from startheight
#btx.rescan=0
## This is the RPC Connection to your node
#ltc.rpc.url=http://127.0.0.1:9332/
#By user name and password
#ltc.rpc.user=bitcoinuser
#ltc.rpc.password=bitcoinpassword
#By cookie file
#ltc.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#ltc.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#ltc.node.endpoint=127.0.0.1:LTC

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#ltc.startheight=-1
## rescan forces a rescan from startheight
#ltc.rescan=0
## This is the RPC Connection to your node
#doge.rpc.url=http://127.0.0.1:22555/
#By user name and password
#doge.rpc.user=bitcoinuser
#doge.rpc.password=bitcoinpassword
#By cookie file
#doge.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#doge.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#doge.node.endpoint=127.0.0.1:DOGE

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#doge.startheight=-1
## rescan forces a rescan from startheight
#doge.rescan=0
## This is the RPC Connection to your node
#bch.rpc.url=http://127.0.0.1:8332/
#By user name and password
#bch.rpc.user=bitcoinuser
#bch.rpc.password=bitcoinpassword
#By cookie file
#bch.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#bch.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#bch.node.endpoint=127.0.0.1:BCH

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#bch.startheight=-1
## rescan forces a rescan from startheight
#bch.rescan=0
## This is the RPC Connection to your node
#grs.rpc.url=http://127.0.0.1:1441/
#By user name and password
#grs.rpc.user=bitcoinuser
#grs.rpc.password=bitcoinpassword
#By cookie file
#grs.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#grs.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#grs.node.endpoint=127.0.0.1:GRS

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#grs.startheight=-1
## rescan forces a rescan from startheight
#grs.rescan=0
## This is the RPC Connection to your node
#btg.rpc.url=http://127.0.0.1:8337/
#By user name and password
#btg.rpc.user=bitcoinuser
#btg.rpc.password=bitcoinpassword
#By cookie file
#btg.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#btg.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#btg.node.endpoint=127.0.0.1:BTG

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#btg.startheight=-1
## rescan forces a rescan from startheight
#btg.rescan=0
## This is the RPC Connection to your node
#dash.rpc.url=http://127.0.0.1:9998/
#By user name and password
#dash.rpc.user=bitcoinuser
#dash.rpc.password=bitcoinpassword
#By cookie file
#dash.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#dash.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#dash.node.endpoint=127.0.0.1:DASH

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#dash.startheight=-1
## rescan forces a rescan from startheight
#dash.rescan=0
## This is the RPC Connection to your node
#polis.rpc.url=http://127.0.0.1:24127/
#By user name and password
#polis.rpc.user=bitcoinuser
#polis.rpc.password=bitcoinpassword
#By cookie file
#polis.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#polis.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#polis.node.endpoint=127.0.0.1:POLIS

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#polis.startheight=-1
## rescan forces a rescan from startheight
#polis.rescan=0
## This is the RPC Connection to your node
#mona.rpc.url=http://127.0.0.1:9402/
#By user name and password
#mona.rpc.user=bitcoinuser
#mona.rpc.password=bitcoinpassword
#By cookie file
#mona.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#mona.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#mona.node.endpoint=127.0.0.1:MONA

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#mona.startheight=-1
## rescan forces a rescan from startheight
#mona.rescan=0
## This is the RPC Connection to your node
#ftc.rpc.url=http://127.0.0.1:9337/
#By user name and password
#ftc.rpc.user=bitcoinuser
#ftc.rpc.password=bitcoinpassword
#By cookie file
#ftc.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#ftc.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#ftc.node.endpoint=127.0.0.1:FTC

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#ftc.startheight=-1
## rescan forces a rescan from startheight
#ftc.rescan=0
## This is the RPC Connection to your node
#ufo.rpc.url=http://127.0.0.1:9888/
#By user name and password
#ufo.rpc.user=bitcoinuser
#ufo.rpc.password=bitcoinpassword
#By cookie file
#ufo.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#ufo.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#ufo.node.endpoint=127.0.0.1:UFO

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#ufo.startheight=-1
## rescan forces a rescan from startheight
#ufo.rescan=0
## This is the RPC Connection to your node
#via.rpc.url=http://127.0.0.1:5222/
#By user name and password
#via.rpc.user=bitcoinuser
#via.rpc.password=bitcoinpassword
#By cookie file
#via.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#via.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#via.node.endpoint=127.0.0.1:VIA

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#via.startheight=-1
## rescan forces a rescan from startheight
#via.rescan=0
## This is the RPC Connection to your node
#xmcc.rpc.url=http://127.0.0.1:24156/
#By user name and password
#xmcc.rpc.user=bitcoinuser
#xmcc.rpc.password=bitcoinpassword
#By cookie file
#xmcc.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#xmcc.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#xmcc.node.endpoint=127.0.0.1:XMCC

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#xmcc.startheight=-1
## rescan forces a rescan from startheight
#xmcc.rescan=0
## This is the RPC Connection to your node
#gbx.rpc.url=http://127.0.0.1:12454/
#By user name and password
#gbx.rpc.user=bitcoinuser
#gbx.rpc.password=bitcoinpassword
#By cookie file
#gbx.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#gbx.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#gbx.node.endpoint=127.0.0.1:GBX

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#gbx.startheight=-1
## rescan forces a rescan from startheight
#gbx.rescan=0
## This is the RPC Connection to your node
#colx.rpc.url=http://127.0.0.1:51473/
#By user name and password
#colx.rpc.user=bitcoinuser
#colx.rpc.password=bitcoinpassword
#By cookie file
#colx.rpc.cookiefile=yourbitcoinfolder/.cookie
#By raw authentication string
#colx.rpc.auth=walletuser:password

## This is the connection to your node through P2P
#colx.node.endpoint=127.0.0.1:COLX

## startheight defines from which block you will start scanning, if -1 is set, it will use current blockchain height
#colx.startheight=-1
## rescan forces a rescan from startheight
#colx.rescan=0
## Disable cookie, local ip authorization (unsecured)
#noauth=0
noauth=1
## What crypto currencies is supported
#chains=btc,btx,ltc,doge,bch,grs,btg,dash,polis,mona,ftc,ufo,via,xmcc,gbx,colx
chains=btc
## Activate or disable verbose logs
#verbose=0
verbose=1

####Server Commands####
#port=24444
port=24444
#bind=127.0.0.1
bind=127.0.0.1
#mainnet=1
testnet=1

####Azure Service Bus####
## Azure Service Bus configuration - set connection string to use Service Bus. Set Queue and / or Topic names to publish message to queues / topics
#asbcnstr=Endpoint=sb://<yourdomain>.servicebus.windows.net/;SharedAccessKeyName=<your key name here>;SharedAccessKey=<your key here>
#asbblockq=<new block queue name>
#asbtranq=<new transaction queue name>
#asbblockt=<new block topic name>
#asbtrant=<new transaction topic name>

NicolasDorier commented 5 years ago

Damn, would you have visual studio and able to debug it? Is it tx that is null?

I reviewed the code and don't see how it can happen.

NicolasDorier commented 5 years ago

Do you have any exception before this?

djpnewton commented 5 years ago

I dont see any other exception in the log.

Yes it is 'tx' that is null (the line tx.RemoveKey($"{_Suffix}IndexProgress", "");), i have breakpointed that line with VS:

        public Task SetIndexProgress(BlockLocator locator)
        {
            return _TxContext.DoAsync(tx =>
            {
                if (locator == null)
                    tx.RemoveKey($"{_Suffix}IndexProgress", "");
                else
                    tx.Insert($"{_Suffix}IndexProgress", "", locator.ToBytes());
                tx.Commit();
            });
        }

NicolasDorier commented 5 years ago

@djpnewton I appreciate. So normally the tx get instantiated on https://github.com/dgarage/NBXplorer/blob/master/NBXplorer/DBriizeTransactionContext.cs#L42 so if you can tell me what is going on it would be cool!

I am surprised that this fail, as normally it would prevent nbxplorer of starting at this point.

djpnewton commented 5 years ago

While that line is being awaited the tx.RemoveKey($"{_Suffix}IndexProgress", ""); line gets executed.

So perhaps it is a race condition that is only showing up on my pc

NicolasDorier commented 5 years ago

@djpnewton possible but I don't see how.

The requests are queued (FIFO) to be executed on a specific thread. And the first request is this which should set the _Tx. One moment I will try something.

NicolasDorier commented 5 years ago

Can you pull and give a try on latest version, I tried to improve logs on wtf going on https://github.com/dgarage/NBXplorer/commit/5b4c62a30fe3864d9f86d25426cd4a5b827f665f

djpnewton commented 5 years ago

The first item in the queue is intended to be that request to set the _Tx but if the rescan flag is set the first item in the queue is placed by SetIndexProgress during the RepositoryProvider constructor

https://github.com/dgarage/NBXplorer/blob/5b4c62a30fe3864d9f86d25426cd4a5b827f665f/NBXplorer/Repository.cs#L79

RepositoryProvider.StartAsync (which ultimately starts the loop an queues the task to set _Tx) cant execute before the contructor and is not called within it

@NicolasDorier are you able to pole holes in my reasoning?

NicolasDorier commented 5 years ago

Thanks for your help, you seem to be right. Why the fuck I could not reproduce on my machine. Working on it.

NicolasDorier commented 5 years ago

Can you try my last commit and close this issue if this fix it?

djpnewton commented 5 years ago

thanks that fixes it 👍

dgarage / NBXplorer

Exception when trying to rescan (a second time) #174