Summary of Progress & Discussion

[who-biz]

Current Progress

In its current state, as of commit https://github.com/who-biz/qwerty-lmdb/commit/870c1af09a42e5edb46a09b18cf55fe013934dd1, this repository builds and will begin syncing if the daemon is launched with: ./src/qwertycoind --db-type=lmdb --log-level=4. Sync progesses until block 391 is encountered. Something causes an error there, and sync can't progress. Still diagnosing where this is coming from.

I also run into the following error, just prior: Block 2e12949a30e9c7991031a4d506f7bd883c56876c0a2b2bee97c80c6a272c6efa already exists in database.

See this commit for the full error message: https://github.com/who-biz/qwerty-lmdb/commit/c5ab41ffbc49bdb90e7b4e2f43d589df04acb316

Implementation, So Far

Structure

• Lmdb's functional components are located in lib/BlockchainDB/Lmdb/db_lmdb.cpp. BlockchainLMDB is implemented as a base class to BlockchainDB.
• In the future, another database structure is preferred, like BerkeleyDB, it can be implemented as a base class of BlockchainDB. similar to LMDB.
• The work was a near-full rewrite of Monero's implementation. Structurally, much is the same.
• Once complete, the user will have the option of storing in the ~/.Qwertycoin/blockchain.bin format, or in the database ~/.Qwertycoin/lmdb/data.mdb. When one is selected, the other one is not constructed or used.
• Most exisiting functions have been edited with condition-enclosed values, which look to the function Tools::getDefaultDbType() in Common/Util.cpp. This function will be modified to check the command line for user input to the --db-type startup flag. However, connection to the startup flag is not yet complete. If you wish to run the daemon using the old format, remove = "lmdb"; from this line: https://github.com/who-biz/qwerty-lmdb/blob/lmdb/lib/Common/Util.cpp#L352
• The calls to DB have been patched into the UpgradeDetector class, for compatibility with the old format. This means the hardfork class is not used, aside from being initialized. As a result, it will likely be removed.
• There are a few excess functions that are not used in Blockchain.cpp and Core.cpp. These will be cut out too, if no specific use can be made of them.
• Exceptions are thrown from the database on fatal errors only. Within those files, there are many conditionals with throw1 lines commented out. Those should be warnings. I didn't have the time to learn too much about the existing logging, so if those are desired - those lines are the locations to add them.
• Currently, the daemon uses a blockchaincache file as well as data.mdb and mdb.lock. These files are all created and saved in the lmdb subdir of the data directory.
• DB drivers and boost modifications are located in the external directory.
• Other boost modifications for binary archives are located in BlockchainDB/binary_archive.h
• All of the new serialization templates are located in BlockchainDB/BlobDataType.h and are enclosed in the serial and detail namespaces.
• Monero-style blobs are patched into the existing blob calls in CryptoNoteCore/CryptoNoteFormatUtils.cpp

[who-biz]

This branch can be compiled to load the blockchain.bin format: https://github.com/who-biz/qwerty-lmdb/tree/blockchain.bin. This format will sync properly, in the current state.

The default branch will compile for LMDB. You'll notice the only difference between these two branches are the function body to Tools::getDefaultDbType().

The LMDB branch still needs fixed with the transaction calls. It looks like the transaction pool, and transaction-related functions in LMDB need added in some more spots. But if you check out the commit history from today, you'll notice that I've begun filling these in. For the time being, I've disabled batch syncing completely on the LMDB branch, while I work out the transaction/key image issues.

[who-biz]

Issue with the key images and transactions appears to be within the Tx pool and moving those transactions into the DB with populated txpool_tx_meta_t struct. That will take a little bit of work, but at least block addition to the DB is working properly, and the blockchain.bin format can still be used as well (something @Xecute0101 mentioned that he definitely wanted to maintain).

[who-biz]

Update Oct 8th, 2019:

As of commit https://github.com/who-biz/qwerty-lmdb/commit/165689a0e17bab722ec4828d53d8829fef588826, sync is tested and confirmed to move past block 391, when using the DB. The database no longer needs the blockchaincache.bin file, when syncing as of this commit: https://github.com/who-biz/qwerty-lmdb/commit/7c2e5e641bec63ac287b74d1c35d2f430690ed4e. This only works properly on first-run, so the data directory needs emptied prior to running in the LMDB format. A second run will load the DB correctly, but as of yet is unable to move past the initial loading. Additionally, sync is somewhat slow, but both of these issues will be ironed out once transactions are working.

As I initially mentioned above, I've now confirmed the issue with key images is occurring in the mempool. Transactions are not being added to the database, due to the transaction pool code still needing updated to accomodate for LMDB. However, all of these functions exist already on the LMDB side, and in Blockchain.cpp, so it is just a matter of adding some conditionals to the functions' in TransactionPool.cpp, and to a lesser extent, Core.cpp.

So, work left to do goes something like this (in order of priority): 1 - Transaction Pool additions 2 - Speed up slow syncing behavior 3 - Fix sync for blockchain.bin format

Outside of the transaction issue, it appears all block-related DB functions are working properly. Somewhere, I messed up the ability to sync in blockchain.bin format, so I'll need to fix that once I'm done working on the database-specific stuff.

[who-biz]

Closing. Will open an issue with updated information.