mvandeberg
commented
6 years ago The blockchain directory should have a special store for saving reversible blocks on shutdown and re-pushing them on startup. Because it does not, the starting block in run (b) is less than the starting block in run (a)
I do not view this as a bug. This is 100% intentional. The block_db implementation of BitShares 2.0 and early versions of Steemd included reversible blocks. It was considerably more buggy, especially during unclean shutdown. The block log is not perfect, but it is considerably more reliable than previous implementations. I recommend this behavior simply be a quirk of running a test net node and no additional development be done. It is obviously not an issue on main net. The only time in which it could be a problem is on a chain halt. The blockchain makes no guarantees about the inclusion of a particular block or transaction. Losing the head block due to reversibility on a halt is no worse than losing pending transactions.
The fact that run (c) started at block 1 means steemd apparently does not check for the situation where block_log exists but shared_memory.bin does not. One reasonable response is to automatically replay in this situation. Another reasonable response is to exit(1) if the user did not specify --replay-blockchain or --resync-blockchain. A less reasonable, but still "correct", response is to rm block_log and resync. A totally buggy, incorrect response is the current behavior, which is apparently to start overwriting block_log without deleting/truncating it.
I agree this should be changed. Instead of automatically replaying, let's print a message prompting for a replay and exit.
The fact that run (d) successfully replayed is, at first glance, not too surprising since presumably the new blocks are the same size. But consider the block times: Blocks 1-16 were irreversible in run (c), so they should have been written into the block log by run (c). But blocks after c's LIB (which cannot be greater than 41, since (c) only produced 41 blocks) can only come from run (b) or possibly run (a). Which means they would have earlier times. Which means run (d) should have failed due to non-increasing timestamps. Which it did not. Which means I can't explain why run (d) didn't fail during replay.
if( log_head_num < dpo.last_irreversible_block_num )The new chain would have appended new blocks past 135 on the new chain. This is problematic because the block log would contain blocks from two different blockchains. However, the replay check for a non-existent shared_memory.bin will prevent this from occurring.
theoreticalbts
steemdproduces blocks 1-145steemdproduces blocks 125-156 (expected, as first shutdown throws away non-irreversible blocks)rm -f shared_memory.bincurlto checkdatabase_api.get_dynamic_global_properties, it returned{"head_block_number":37, "last_irreversible_block_num":16}steemd --replay-blockchainproduces blocks 136-147That this sequence of events can occur reveals several bugs:
steemdapparently does not check for the situation whereblock_logexists butshared_memory.bindoes not. One reasonable response is to automatically replay in this situation. Another reasonable response is toexit(1)if the user did not specify--replay-blockchainor--resync-blockchain. A less reasonable, but still "correct", response is torm block_logand resync. A totally buggy, incorrect response is the current behavior, which is apparently to start overwritingblock_logwithout deleting/truncating it.Without looking at the code, my best guess is that it's some race condition with the OS filesystem. So for reference, here is the environment (running on a manually provisioned AWS EC2 instance):
Here's the log:
In another terminal window, result of
get_dynamic_global_propertiesduring the second execution which produced from block 1: