Segfault when loading wallet

[sturles]

I tried to compile btc1 Core from git, and use it with my wallet.dat from Bitcoin Core on my backup bitcoin node, but it doesn't work. It segfaults on loading the wallet. I assume the wallet formats are supposed to be compatible, and this is a bug.

Version is: btc1 Core Daemon version v1.14.5.0-a301696

Running under Ubuntu 16.04.3 LTS on an Odroid-XU4: Linux luna 3.10.105 #1 SMP PREEMPT Tue Mar 28 13:17:20 UTC 2017 armv7l armv7l armv7l GNU/Linux

The wallet.dat was last copied from Bitcoin Core 0.14.2-UASF running on AMD64. It was created in 2010, and it works fine on Bitcoin Core 0.12 on the same backup node.

The last few lines from debug.log:

2017-08-07 00:16:30 Initializing databases...
2017-08-07 00:16:30 init message: Verifying blocks...
2017-08-07 00:16:30  block index           13640ms
2017-08-07 00:16:30 init message: Loading wallet...

gdb says:

Thread 1 "bitcoind" received signal SIGSEGV, Segmentation fault.
0xb6bacbae in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
(gdb) where
#0  0xb6bacbae in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#1  0x7f70a642 in std::_Rb_tree_iterator<std::pair<uint256 const, CWalletTx> >::operator-- (this=<synthetic pointer>) at /usr/include/c++/5/bits/stl_tree.h:220
#2  std::_Rb_tree<uint256, std::pair<uint256 const, CWalletTx>, std::_Select1st<std::pair<uint256 const, CWalletTx> >, std::less<uint256>, std::allocator<std::pair<uint256 const, CWalletTx> > >::_M_get_insert_hint_unique_pos (__k=..., 
    __position=..., this=0x85b7a654) at /usr/include/c++/5/bits/stl_tree.h:1924
#3  std::_Rb_tree<uint256, std::pair<uint256 const, CWalletTx>, std::_Select1st<std::pair<uint256 const, CWalletTx> >, std::less<uint256>, std::allocator<std::pair<uint256 const, CWalletTx> > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<uint256 const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<uint256 const, CWalletTx> >, std::piecewise_construct_t const&, std::tuple<uint256 const&>&&, std::tuple<>&&) (this=0x85b7a654, __pos=...)
    at /usr/include/c++/5/bits/stl_tree.h:2174
#4  0x7f721ce0 in std::map<uint256, CWalletTx, std::less<uint256>, std::allocator<std::pair<uint256 const, CWalletTx> > >::operator[] (this=0x85b7a654, 
    __k=...) at /usr/include/c++/5/bits/stl_map.h:483
#5  0x7f711362 in CWallet::LoadToWallet (this=this@entry=0x85b7a4c0, wtxIn=...)
    at wallet/wallet.cpp:1004
#6  0x7f72eed0 in ReadKeyValue (pwallet=pwallet@entry=0x85b7a4c0, ssKey=..., 
    ssValue=..., wss=..., strType="tx", strErr="") at wallet/walletdb.cpp:335
#7  0x7f72ff18 in CWalletDB::LoadWallet (this=this@entry=0xbeffdd98, 
    pwallet=pwallet@entry=0x85b7a4c0) at wallet/walletdb.cpp:596
#8  0x7f714cea in CWallet::LoadWallet (this=this@entry=0x85b7a4c0, 
    fFirstRunRet=@0xbeffde07: false) at wallet/wallet.cpp:2848
#9  0x7f71cbf0 in CWallet::CreateWalletFromFile (walletFile="wallet.dat")
    at wallet/wallet.cpp:3602
#10 0x7f71db86 in CWallet::InitLoadWallet () at wallet/wallet.cpp:3769
#11 0x7f593ce0 in AppInitMain (threadGroup=..., scheduler=...) at init.cpp:1564
#12 0x7f57caca in AppInit (argc=1, argv=<optimized out>) at bitcoind.cpp:167
#13 0x7f572994 in main (argc=1, argv=0xbefffbd4) at bitcoind.cpp:196

[christophebiocca]

When you compiled btc1 from source, which flags did you use (with ./configure)?

Did you compile Core 0.12 from source as well or use the gitian builds?

The reason I'm asking is because differing BDB versions may very well lead to a segfault.

[sturles]

Both are configured with --with-incompatible-bdb and compiled from source using the same version of BDB (5.3) on the same machine. 0.14.2-UASFis configured in the same way, using the same version of BDB, but on a different architecture.

[sturles]

Result of valgrind ./bitcoind is here: https://pastebin.com/Fpiqp6sb

[christophebiocca]

I'm looking into that code more closely. Can you tell me which exact git commit you compiled with?

[sturles]

@christophebiocca It is part of the version string:
btc1 Core Daemon version v1.14.5.0-a301696

[christophebiocca]

Unfortunately operator[] is never supposed to error during normal operation (it will create nonexistent entries as needed). So this is likely due to hitting UB before even getting to that line.

Here's a patch that prints out what kinds of data it's loading (and how many records it's read so far), which should help us find the underlying problem.

index 81fdde401..be8fd274c 100644
--- a/src/wallet/walletdb.cpp
+++ b/src/wallet/walletdb.cpp
@@ -287,6 +287,9 @@ ReadKeyValue(CWallet* pwallet, CDataStream& ssKey, CDataStream& ssValue,
         // Taking advantage of the fact that pair serialization
         // is just the two items serialized one after the other
         ssKey >> strType;
+
+        LogPrintf("Loading object of type %s\n", strType);
+
         if (strType == "name")
         {
             string strAddress;
@@ -558,6 +561,7 @@ DBErrors CWalletDB::LoadWallet(CWallet* pwallet)
     CWalletScanState wss;
     bool fNoncriticalErrors = false;
     DBErrors result = DB_LOAD_OK;
+    int recordIndex = 0;

     LOCK(pwallet->cs_wallet);
     try {
@@ -591,6 +595,8 @@ DBErrors CWalletDB::LoadWallet(CWallet* pwallet)
                 return DB_CORRUPT;
             }

+            LogPrintf("Reading next record (%d) from wallet db.\n", recordIndex++);
+
             // Try to be tolerant of single corrupt records:
             string strType, strErr;
             if (!ReadKeyValue(pwallet, ssKey, ssValue, wss, strType, strErr))

[sturles]

Looks like it segfaults on the first transaction it sees in my wallet:

2017-08-07 22:28:18 init message: Loading wallet...
2017-08-07 22:28:18 Reading next record (0) from wallet db.
2017-08-07 22:28:18 Loading object of type tx

[christophebiocca]

If you're ok with it I think the next step will be to dump that transaction's exact contents so that we can try to recreate the issue locally. This should help us narrow down which part is triggering UB issues, and rule out issues related to custom BDB library/compiler by running the same code in the standard configuration.

I should have a patch ready for this in the evening.

[nomnombtc]

You said the wallet was copied from an amd64 machine, maybe it would be a good test to compile bitcoin-core 0.14.2 on this arm machine with the same options and see if it also crashes with the wallet.dat.

[sturles]

@nomnombtc It does, of course. 0.12 does not.

[nomnombtc]

Well if that also crashes it at least rules out the changes that have been done for segwit2x. Maybe try if you can dump the contents of wallet.dat on the commandline db_dump wallet.dat, or if that also gives some error. But whatever is causing this should probably not result in a segfault :/

[sturles]

I have already tried to db5.3_dump my wallet.dat, and db4.8_load into a wallet.dat for libdb 4.8. This works, but attempts to load it into bitcoind linked with libdb4.8 segfaults in the same way. Therefore I don't think it is a problem with libdb.

[sturles]

Is there any progress on fixing this bug?

[bitPico]

There is no bug. You broke your wallet file by moving it onto a system that does not support it. Read the BDB documentation.

[sturles]

@bitPico Read my comments. I have already eliminated libdb as a possible cause by dumping on the original platform and loading into different libdb versions on the ARM.