6 out 8 masternodes crashed

[fogoat]

Okdoki on discord reporting many masternodes crashed

6/8 MN crashed just after block 104899 accepted @ 2019-03-03 12:32:29 2/8 MN were still active after block 104899, here is the log of one of them:

019-03-03 11:40:25 CGovernanceManager::ProcessVote -- Unknown parent object , MN outpoint = 22b532dd5df9e171ddce49733b4bc7ff88e6a5fbddbf34aacee61ff361cdb5f5-0, governance object hash = a700f834e4392e099b4948251af3af505$
2019-03-03 11:40:40 DSEG -- Sent 1 Masternode inv to peer 2671
2019-03-03 11:40:40 DSEG -- Sent 1 Masternode inv to peer 2660
2019-03-03 11:55:18 CGovernanceManager::UpdateCachesAndClean -- erase obj d427d016dfaa31a65b3424c71bd85eef7bbe82ee8e62dd84f66499fa85aa6d8e
2019-03-03 11:55:22 CGovernanceManager::ProcessVote -- Unknown parent object , MN outpoint = c7c6c4a048bf1ee3bfd2db73fa078a5ca6d291993835d2ac08364fe56f64b677-1, governance object hash = d427d016dfaa31a65b3424c71bd85eef7$
2019-03-03 12:07:31 AcceptToMemPool::Tithe Overflow Type I; Tithe Rejected; Tithe Count 76, Pog Diff 2296.026167 AcceptToMemPool::Tithe Overflow Type I; Tithe Rejected; Tithe Count 76, Pog Diff 2296.026167 AcceptToMemPo$
2019-03-03 12:19:01 CMasternode::UpdateLastPaidBlock -- searching for block with payment to fe35a0f028a316671e678203d5f3433f6d76423a0cd53d9e31ee6e4e7129784d-1 -- found new 104895.000000
2019-03-03 12:19:01 ProcessNewBlock : ACCEPTED
2019-03-03 12:25:39 UpdateTip: new best=3ab5bb6c0a70b257de423ae2740a8734734ea0074337da0cfd43564ce9d00d2e  height=104896  log2_work=59.185375  tx=877606  date=2019-03-03 12:25:26 progress=1.000000  cache=1.2MiB(618tx)
2019-03-03 12:25:39 CMasternode::UpdateLastPaidBlock -- searching for block with payment to 48824b64141ddaa2ebeda33318c5e4936474bfcd9bbdf431c70d30d1ddf68daf-1 -- found new 104896.000000
2019-03-03 12:25:39 ProcessNewBlock : ACCEPTED
2019-03-03 12:26:24 UpdateTip: new best=a21f87a3e377a55861e083fdac4a27f370dbd6710fe37d17a6dbc5a85018ab87  height=104897  log2_work=59.185375  tx=877614  date=2019-03-03 12:26:27 progress=1.000000  cache=1.2MiB(631tx)
2019-03-03 12:26:24 CMasternode::UpdateLastPaidBlock -- searching for block with payment to 48824b64141ddaa2ebeda33318c5e4936474bfcd9bbdf431c70d30d1ddf68daf-1 -- found new 104897.000000
2019-03-03 12:26:24 ProcessNewBlock : ACCEPTED
2019-03-03 12:28:38 UpdateTip: new best=a7f23e3a7d40c8558f1982776aee5bcd8f3641de60f3c0dfb734d1f1dc534c0c  height=104898  log2_work=59.185376  tx=877626  date=2019-03-03 12:28:31 progress=1.000000  cache=1.3MiB(669tx)
2019-03-03 12:28:38 CMasternode::UpdateLastPaidBlock -- searching for block with payment to 43d563990057e02ee52bb9ece2555c8c77586de4dbb08c626a7266d4c3363d84-1 -- found new 104898.000000
2019-03-03 12:28:38 ProcessNewBlock : ACCEPTED
2019-03-03 12:30:01  CheckingHealth... LastBlockAge 90, LastAcceptBlockAge 83, Diff 571.130553 ...
HealthCheckup::Healthy.
2019-03-03 12:32:29 UpdateTip: new best=e96785177c315083249c23077d6223fa6d6accf5827fd9709dca5982f487f9db  height=104899  log2_work=59.185377  tx=877638  date=2019-03-03 12:31:59 progress=0.999999  cache=1.3MiB(695tx)
2019-03-03 12:32:29 CMasternode::UpdateLastPaidBlock -- searching for block with payment to 0503595975b14afd86b83d527b61b0f30dcd200329ee301ee4841db116b55f60-1 -- found new 104899.000000
2019-03-03 12:32:29 ProcessNewBlock : ACCEPTED
2019-03-03 12:33:34
Unable to find hashBlock 0000000000000000000000000000000000000000000000000000000000000000
Unable to find hashBlock 0000000000000000000000000000000000000000000000000000000000000000
Unable to find hashBlock 0000000000000000000000000000000000000000000000000000000000000000
Unable to find hashBlock 0000000000000000000000000000000000000000000000000000000000000000
Unable to find hashBlock 0000000000000000000000000000000000000000000000000000000000000000
Unable to find hashBlock 0000000000000000000000000000000000000000000000000000000000000000DSEG -- Sent 1 Masternode inv to peer 1479
2019-03-03 12:39:32 UpdateTip: new best=2792af5e1cc4d3d69a181cf6bf22c1b734f19445d3fdb66c63a878a4a3ece215  height=104900  log2_work=59.185379  tx=877675  date=2019-03-03 12:39:23 progress=1.000000  cache=1.3MiB(759tx)
2019-03-03 12:39:32 CMasternode::UpdateLastPaidBlock -- searching for block with payment to a234e978b5b9683e19f7f5fd24228e801d3e9bf10b3fc79a65043dab90b764ef-1 -- found new 104900.000000
2019-03-03 12:39:33 ProcessNewBlock : ACCEPTED
2019-03-03 12:44:48 UpdateTip: new best=29e6c6f559856c14c515bd54a5bae8d502cde69e7aa6b93f4a557bbea493ad3f  height=104901  log2_work=59.185381  tx=877690  date=2019-03-03 12:44:28 progress=0.999999  cache=1.3MiB(782tx)
2019-03-03 12:44:48 CMasternode::UpdateLastPaidBlock -- searching for block with payment to 3a11a43278bce7048160248a44d13cb499fb69affaee96973e2d145500e9f683-1 -- found new 104901.000000
2019-03-03 12:44:48 ProcessNewBlock : ACCEPTED
2019-03-03 12:48:18 UpdateTip: new best=2912d991e54332e2e7adc7f392031064b9aa7f611fcc1d2bf644ead3c081cb54  height=104902  log2_work=59.185382  tx=877703  date=2019-03-03 12:48:15 progress=1.000000  cache=1.3MiB(801tx)
2019-03-03 12:48:18 CMasternode::UpdateLastPaidBlock -- searching for block with payment to 4ea90825ba476204a2d7e783f297472c7459d7c89c8a9842494756a9470342a1-1 -- found new 104902.00000

Once again this Unable to find hashBlock 0000000000000000000000000000000000000000000000000000000000000000 line...

[biblepay]

Please run the sanctuary in valgrind from a compiled copy, then enter the issue with the valgrind log after proof of the crash is found.

[biblepay]

Yes, I will keep this open, please, give us a valgrind log.

[fogoat]

Attached is the valgrind log from a MN that crashed. I hope this helps.

6of8-crashed-valgrind-output.log

[fogoat]

Here is the second crash from different node.

6of8-second-crash.log

[biblepay]

@sunk818 Ok, I took a look at the valgrind log.

It appears whoever ran it didnt follow the instructions above - was it run from a biblepay binary, or a compiled biblepay node? I mentioned that valgrind must be run on a compiled copy.

Log is not helpful. The other log is the same as the first log - binary.

[vahtis]

Could you provide better instructions. You are asking help from people who are not familiar with compiling etc

[biblepay]

Sure! Basically, what we are trying to do is trap an error at runtime and capture the specific original line number and c++ filename, since we believe this particular error is in an untraceable procedure most likely occurring in the middle of a legacy instantsend transaction. The reason we need the exact line, is because c++ casts resulting in segfaults could be anywhere, and therefore its almost impossible to put enough logging in to catch the exact line without valgrind. In order for valgrind to output anything useful, it needs the original symbols encoded and the debug line number pointers. It can only get that from a running biblepay version that was compiled on the same flavor box as it is running (IE ubuntu 16.04 is running a biblepay compiled on ubuntu 16.04 64 bit version by gnu, etc).

So, in light of this, all you have to do is compile biblepay on the machine with valgrind, then run biblepay in valgrind.

To compile bbp just use the existing instructions: https://github.com/biblepay/biblepay/blob/master/BuildBiblePay.txt

After compiled if you are running QT for instance:

cd ~/biblepay/src/qt Instead of launching it normally like this: ./biblepay-qt

Launch like this:

valgrind ./biblepay-qt

Of course we want to capture the output to the log, so please do the same as you did before (capture both stdout and stderr, as valgrind prints out very extremely important info to stderr) and we would probably miss what we need if both pipes are not redirected to the log.

[biblepay]

One more thing; If you crash a lot, I recommend cleaning your .dat files and blocks and chainstate and resyncing. The crash could be from having corrupted mncache or mnpayment data.

[fogoat]

I believe this is no longer an issue since you upgraded all your masternodes and the consensus conflict is gone with over 50% agreeing on version 1198 code now. Will re-open if Discord user reports further issues.