search

dashpay / dash

Dash - Reinventing Cryptocurrency
https://www.dash.org
MIT License
1.49k stars 1.2k forks source link

Unexpected Transition to EXPIRED - 12.1 TestNet #1236

Closed ryansd92 closed 6 years ago

ryansd92 commented 7 years ago

@tgflynn and I were working through an earlier issue (https://github.com/dashpay/dash/issues/1233) that this might be related to.

Last night, I was able to get 5 masternodes and 1 Mac OS wallet running by issuing the following command:

invalidateblock 000000007feb34b9d83bd883104f6ea9e69a626a54fe352f58422d4b69e1d91c

Although this enabled me to get all masternodes and wallet on the correct chain, all masternodes eventually went offline. The sequence of events was:

  1. Log into each masternode and issue the command above
  2. Open Dash GUI on Mac OS and issue the command above from the debug console.
  3. Issue Start All from GUI
  4. 11:14pm ET: All masternodes transition to PRE_ENABLED
  5. 11:24pm ET: 2 masternodes transition to ENABLED
  6. 11:29pm ET: 1 masternodes transitions to ENABLED
  7. 11:34pm ET: Remaining 2 masternodes transition to ENABLED
  8. 12:49am ET: 1 masternode transitions to EXPIRED
  9. 12:54am ET: 1 masternode transitions to EXPIRED
  10. 12:59am ET: 2 masternodes transition to EXPIRED
  11. 2:44am ET: 1 masternode transitions to NEW_START_REQUIRED
  12. 2:49am ET: 1 masternode transitions to NEW_START_REQUIRED
  13. 2:54am ET: 2 masternodes transition to NEW_START_REQUIRED
  14. (As of this report) 1 masternode remains ENABLED

My GUI Wallet shows a status for all masternodes as NEW_START_REQUIRED. The wallet is not on the correct chain anymore.

As you can see from the screenshot, 4 of the 5 masternodes appear to be on the wrong chain. The one that is correct (2nd from the left) is the one that never went offline.

Why do my masternodes keep falling off the right chain?

screen shot 2016-12-30 at 8 09 04 am
UdjinM6 commented 7 years ago

So, to clarify, transition itself is ok actually: masternodes are not on the right chain and so they are not able to send correct messages and then they finally expired. The real issue is why they stuck on some another chain and do not switch to the right one. This probably can be answered by looking at debug.log files of these masternode. Can you provide one? ---email removed---

ryansd92 commented 7 years ago

@UdjinM6, yes. I understand. The transitions to other states is related to the bad chain. I have all the log files. Give me a few minutes to get them. I will upload them here.

ryansd92 commented 7 years ago

Here is one log file. I have the others if you want them. This one is very small.

debug.tar.gz

UdjinM6 commented 7 years ago

@tgflynn looks like it's "missed trigger" issue again

2016-12-30 04:18:54 ERROR: ConnectBlock(): coinbase pays too much at height 127584 (actual=5750000000 vs limit=2250000000), exceeded block reward, no triggered superblock detected
2016-12-30 04:18:54 Misbehaving: 34.195.141.233:19999 (0 -> 100) BAN THRESHOLD EXCEEDED
2016-12-30 04:18:54 InvalidChainFound: invalid block=0000000548d63d2b73ad2cd1ab64aa88c00f9824f36a8844c09feecc7062634b  height=127584  log2_work=43.14167

@ryansd92 so, a temporary solution to let your nodes sync is this sequence of commands (use dash-cli):

mnsync reset
reconsiderblock 0000000548d63d2b73ad2cd1ab64aa88c00f9824f36a8844c09feecc7062634b
clearbanned
ryansd92 commented 7 years ago

Ok, I will run that on all the nodes now. Thank you.

ryansd92 commented 7 years ago

That was successful. My wallet and all 5 masternodes are all on the right block and fully sync'd. I used a remote start all and each masternode transitioned to PRE_ENABLED. I confirmed this with a separate non-masternode instance. Now I'll wait for them to become ENABLED. I am still running yesterday's binaries. I see there was a build last night. Should I upgrade all nodes to that build?

tgflynn commented 7 years ago

I believe that this along with the problems we saw last night are due to the fact that due to the addition of a huge amount of hash power (probably an ASIC or GPU miner) to one of the testnet mining pools an incorrect chain with higher difficulty exists on testnet. Note that this is a circumstance that would be virtually impossible on mainnet so it is really a testnet only problem.

A new checkpoint has now been added which should force all nodes onto the correct chain. If you upgrade to commit 96dda2790c145b41ef9dccd7e2f4c376385aeb54, I think it should resolve these problems.

schinzelh commented 7 years ago

Latest build contains a new checkpoint to keep your peers on the right chain, so feel free to update.

ryansd92 commented 7 years ago

Thanks, @schinzelh and @tgflynn - I will update to the latest build very soon.

UdjinM6 commented 7 years ago

127584 is quite recent block and it's not related to that huge hashpower spike imo, just a regular missed trigger issue.

tgflynn commented 7 years ago

@UdjinM6 Hmm, I assumed that if all his nodes were affected it must have been due to the other chain.

We haven't seen the missing trigger issue affect multiple nodes at once like that.

UdjinM6 commented 7 years ago

Yep, that what concerns me, asic spike was at 125992. There is a lot of rate check fails in that log, maybe that's somehow related.

tgflynn commented 7 years ago

Just noticed that, they appear to be mostly "too old timestamp" messages. That shouldn't really affect any relevant trigger. I wonder if the chain backtracking caused by the fork is somehow causing this. The code pretty much assumes that block height increases monotonically, like time.

schinzelh commented 6 years ago

Should be fixed, testnet is on version 12.2 now.