Versioning of plugin data

[theoreticalbts]

When plugin settings change, plugin needs to be able to request a re-index. Likewise, some plugins may never need to re-index -- for example, plugins that don't provide objects, or whose object lifetimes are limited to a single run.

Related: #264 proposes updating based on schema changes, this code will have to interact with that.

[mvandeberg]

This is partially implemented in chainbase already but should be formalized by #264

[theoreticalbts]

Specifically, the version of a file may be defined as schema and there is semantics. Schema is the set of available fields.

• Schema: Whenever a field is added anywhere in the database, the binary format becomes invalid and a reindex is necessary.
• Semantics: Whenever a change is made to the algorithm used to calculate a field, the database contents become invalid and a reindex is necessary. For example, consider a statistics plugin which accumulates various historical data in buckets. If the configuration of the plugin is changed so that bucket size or membership changes, then the blockchain must be reindexed.

Plugins and core must create schema and semantics objects. Schema objects can automatically be created with FC (see #683). Semantics objects must be manually defined, and manually updated whenever the way a field is calculated changes. Updating semantics should be made part of the release process for every major release.

[theoreticalbts]

I should make a special note that the core semantics version is different from STEEMIT_BLOCKCHAIN_VERSION. Here are some of the differences:

• By changing bucket sizes, it is possible to create a different semantics with exactly the same code base.
• Fields like low-memory, testnet, clear votes affect the core semantics, but not STEEMIT_BLOCKCHAIN_VERSION.
• It is possible to have a version update that doesn't change the DB structure in any way, but only e.g. refactors code and fixes API bugs. Such an update could have the same core semantics version.

[theoreticalbts]

Let's go for a little more detailed specification of what we should actually implement for this.

• Implement steem_abstract_plugin which subclasses abstract_plugin
• steem_abstract_plugin defines a virtual method virtual optional<sha256> get_state_config_hash()const = 0
• On startup, for each plugin that inherits from steem_abstract_plugin, we call get_state_config_hash() and put all (plugin.get_name(), plugin.get_state_config_hash()) pairs in a vector< pair< string, sha256 > >.
• Finally we add ("core", database.get_state_config_hash())
• Hash the vector
• When creating a new shared memory file, put the hash in a singleton object
• When starting up with an existing shared memory file, compare the hash to the singleton object that was loaded, reindex if there is a mismatch
• The get_state_config_hash() value of "core" or any plugin should be changed whenever there is a data structure change or change of past semantics that invalidates the old version
• The exact method that computes get_state_config_hash() can be anything, the simplest is just H(version) but you could include other information, e.g. for market stats it could include bucket sizes so it reindexes when you change bucket sizes, or if schema plugin is finally completed it could include a hash autogenerated from the data structure definitions, so any change to data structure fields automatically invalidates old files without requiring any explicit version number bump
• We should try to be really careful to change the hash every time we make a change that requires a re-index.