clamd 2.0.0-rc1 crashes on initial block download

[dooglus]

I've been switching between versions of clamd and somehow corrupted my blockchain.

I ended up deleting all the block indexes and trying to start again, using the existing blk000n.dat files as a bootstrap.dat. I made a bootstrap.dat file like this:

$ cd ~/.clam/blocks
$ l
total 1188452
-rw------- 1 chris chris 814304490 Jan 13  2016 blk00000.dat
-rw------- 1 chris chris 134215726 May  9  2016 blk00001.dat
-rw------- 1 chris chris 134217444 Oct  3 10:18 blk00002.dat
-rw------- 1 chris chris 134217728 Jan 14 14:24 blk00003.dat
$ cat blk* > ../bootstrap.dat
$ 

then left clamd processing the file overnight.

In the morning clamd was dead. I tried restarting it, bu after skipping all the blocks it had already processed:

2017-01-18 16:18:22 ERROR: ProcessBlock() : already have block 877065 4985fc1365a4c917f3d4117b192e5ee2c403e0948349f25bc9360985bb685d3f
2017-01-18 16:18:22 ERROR: ProcessBlock() : already have block 877066 d5b31f0bb0b5b5528a68da9131fb899fc33734ee87393403e7f5c7f5e0174fc1
2017-01-18 16:18:22 ERROR: ProcessBlock() : already have block 877067 f0ce35c826a086f78ab9514dd58e8d2c807d90d04b63b34a47ffb52279ec5a63
2017-01-18 16:18:22 ERROR: ProcessBlock() : duplicate proof-of-stake (COutPoint(6155188d80, 1), 1456431472) for block 12d580f986f86ed81726396d1bf099a1926837ec2ee387a091c9be861401acd8
2017-01-18 16:18:22 ProcessBlock: ORPHAN BLOCK 0, 0 MB out of 41943040, prev=12d580f986f86ed81726396d1bf099a1926837ec2ee387a091c9be861401acd8
2017-01-18 16:18:22 ProcessBlock: ORPHAN BLOCK 0, 0 MB out of 41943040, prev=960701219e976360f80100220b50c4031b789842cf1f7a324ae1ea247e2685e7
2017-01-18 16:18:22 ProcessBlock: ORPHAN BLOCK 0, 0 MB out of 41943040, prev=1cc1778ea8791bc35c2bdad1a465c73206900bebdb050d126e1a2fc186f0fd05
2017-01-18 16:18:22 ProcessBlock: ORPHAN BLOCK 0, 0 MB out of 41943040, prev=1a43138e255e62c6d12ada7fa4eadf1f9fab10fda3083b0eabfefa47329773cc

I don't know what was going on there. All the following blocks were seen as 'orphan block 0'.

I stopped it, moved the bootstrap.dat file out of the way, restart it, let it connect to a peer, and that got it past the sticking point. Stopped it again, put the bootstrap.dat file back in place, and it was able to continue loading from it, until:

ProcessBlock() : parent block was previously rejected because of stake duplication. Reaccepting parent

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe75c3700 (LWP 5692)]
CheckStakeKernelHash (pindexPrev=0x0, nBits=453022402, blockFrom=..., nTxPrevOffset=150, txPrev=..., prevout=..., nTimeTx=1457590624, hashProofOfStake=..., 
    targetProofOfStake=..., fPrintProofOfStake=false) at kernel.cpp:401
401     if (IsProtocolV2(pindexPrev->nHeight+1))
(gdb) where
#0  CheckStakeKernelHash (pindexPrev=0x0, nBits=453022402, blockFrom=..., nTxPrevOffset=150, txPrev=..., prevout=..., nTimeTx=1457590624, hashProofOfStake=..., 
    targetProofOfStake=..., fPrintProofOfStake=false) at kernel.cpp:401
#1  0x0000555555695675 in CheckProofOfStake (pindexPrev=pindexPrev@entry=0x0, tx=..., nBits=453022402, hashProofOfStake=..., targetProofOfStake=...)
    at kernel.cpp:436
#2  0x00005555555d3376 in CheckProofOfStake (pindexPrev=pindexPrev@entry=0x0, pblock=pblock@entry=0x7fffe0f61590, hash=..., state=...) at main.cpp:3040
#3  0x00005555555ecf1b in ProcessBlock (state=..., pfrom=pfrom@entry=0x0, pblock=pblock@entry=0x7fffe75c2c20, dbp=dbp@entry=0x0) at main.cpp:3176
#4  0x00005555555edfef in LoadExternalBlockFile (fileIn=fileIn@entry=0x7fffe0000980, dbp=dbp@entry=0x0) at main.cpp:3925
#5  0x00005555555a8b43 in ThreadImport (vImportFiles=std::vector of length 0, capacity 0) at init.cpp:372
#6  0x00005555555c2bb4 in operator()<void (*)(std::vector<boost::filesystem::path>), boost::_bi::list0> (f=<optimized out>, a=<synthetic pointer>, 
    this=<optimized out>) at /usr/include/boost/bind/bind.hpp:253
#7  operator() (this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:20
#8  boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> >), boost::_bi::list1<boost::_bi::value<std::vector<boost::filesystem::path, std::allocator<boost::filesystem::path> > > > > >::run (this=<optimized out>)
    at /usr/include/boost/thread/detail/thread.hpp:117
#9  0x00007ffff753aaea in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0
#10 0x00007ffff5ca40a4 in start_thread (arg=0x7fffe75c3700) at pthread_create.c:309
#11 0x00007ffff59d962d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) p pindexPrev
$1 = (CBlockIndex *) 0x0
(gdb) 

[YiumPotato]

dooglus stop orphaning my blocks smh

[dooglus]

I was hearing that solo-staking was very hard due to Just-Dice's staking wallet orphaning people's blocks, so I set up an experiment.

I made a new wallet and sent 10,000 CLAM to it, split into 100 CLAM pieces.

I wanted to see how well it staked, and how much it would be orphaned by the Just-Dice staking wallet.

Here's the wallet so you can check for yourself how it's getting on.

So far it has been staking for 5.73 days. In that time it has staked 57 times, and of those 57 blocks 6 have been orphaned. That's 10.5% of the staked blocks which were orphaned.

The remaining 51 successful stakes in 5.73 days mean that when solo-staking you need 1123.46 CLAM to stake once per day.

Over the last 5 days Just-Dice has staked 1313.6621 + 1288.7924 + 1323.2905 + 1301.2370 + 1283.4634 = 6510.4454 CLAMs. JD is staking roughly 1575000 CLAMs.

Those numbers suggest that on JD you need 1575000 / 6510.4454 * 5 = 1209.59 CLAM to stake once per day, and so over the last 5 days it has actually been better to solo stake than to stake at JD. That's not even taking JD's 10% staking commission into account.

This is likely the result of short-term variance. I can't believe that it is more efficient to solo-stake than to use the JD wallet. I'll report back in a week or so.

Edit: raw data from the 10k solo-staking wallet:

staked 51 times in 5.73 days; need 1123.46 coins to stake once per day
staked 54 times in 5.84 days; need 1081.01 coins to stake once per day
staked 57 times in 6.00 days; need 1053.30 coins to stake once per day
staked 57 times in 6.52 days; need 1143.49 coins to stake once per day
staked 61 times in 7.48 days; need 1226.05 coins to stake once per day
staked 65 times in 7.96 days; need 1224.15 coins to stake once per day
staked 79 times in 9.43 days; need 1193.23 coins to stake once per day
staked 81 times in 10.10 days; need 1246.65 coins to stake once per day
staked 91 times in 11.35 days; need 1247.38 coins to stake once per day
staked 96 times in 12.33 days; need 1284.26 coins to stake once per day
staked 100 times in 12.94 days; need 1293.59 coins to stake once per day
staked 103 times in 13.50 days; need 1310.49 coins to stake once per day
staked 110 times in 14.46 days; need 1314.56 coins to stake once per day
staked 145 times in 18.33 days; need 1263.87 coins to stake once per day
staked 149 times in 19.31 days; need 1296.09 coins to stake once per day
staked 164 times in 21.29 days; need 1297.97 coins to stake once per day
staked 171 times in 22.42 days; need 1310.90 coins to stake once per day
staked 180 times in 23.36 days; need 1297.83 coins to stake once per day
staked 192 times in 25.00 days; need 1302.24 coins to stake once per day
staked 200 times in 26.34 days; need 1317.12 coins to stake once per day
staked 213 times in 28.31 days; need 1329.07 coins to stake once per day
staked 228 times in 30.29 days; need 1328.37 coins to stake once per day
staked 553 times in 75.68 days; need 1368.55 coins to stake once per day

[YiumPotato]

Thank you @dooglus for always "proving" being trust worthy (even though I was joking before, I still really appreciate your messages)

[dooglus]

I posted mostly because I wanted somewhere to log the results of the experiment I'm running. It's out of place in this issue, but whatever. :)

[haraldg]

I posted mostly because I wanted somewhere to log the results of the experiment I'm running. It's out of place in this issue, but whatever. :)

I'd suggest adding such information to the wiki here on github, where it will stay available even after this issue is closed.

[tryphe]

I've been using this branch for a month on a wallet with no coin in the wallet, and it's not corrupt. The diff compared to master is pretty long, so it's hard to guess where the problem could be.

[l0rdicon]

I'm not entirely sure where the logic error is yet, but the block of code I believe we will find it is @ https://github.com/nochowderforyou/clams/blob/2.0.0-rc1/src/main.cpp#L3075-L3114

[dooglus]

I was switching between different wallet.dat files when I saw the corruption. It often seemed to want to reindex (or rescan, I forget) around half the chain when I switched, even though the wallet had recently been used. I don't know if that's related. The time my wallet got corrupted I had done a hard "kill -9" on the clam process because it was going to take hours to do what seemed to be an unnecessary rescan.

[tryphe]

It looks to me like the issue could be happening due to this commit: c940461

If you look at the same function in the rat4's master branch here there are some comments which suggest not changing chain data in ram at the loading stage:

// Update block index on disk without changing it in memory.
// The memory index structure will be changed after the db commits.

My hunch is that it's not happening for me because there's no stake transactions in my wallet, but I'm not entirely sure.

[l0rdicon]

Closing issue, unrelated to client v2.1.0