Add persistence md

[ghost]

Adds persistence

lots to discuss... I'm not 100% sure the way i made get_prices and get_indexes available to mastercore.cpp is the best way (using mastercore::)

i reused some of the dex variable names (but added comments on how metadex uses them)

Most loading/saving logic came from mastercore_dex.cpp, cleaned up a little

no way to set the action, so added another method to do it (setAction)

[ghost]

commit cb48b0a is RPC-only, let me know if I need to break that out, but it shouldn't affect testing of the branch.

[ghost]

Added lexical sorting

[m21]

Haven't reviewed everything yet guys, but I believe subaction does not even need to be stored in this case. The (original Dex) intent of the offers.dat files is to re-create the in-memory structures needed for trading. Same works for Metadex (actually less info is required for Metadex, it doesn't have ACCEPTs, not blocktimelimit, nor fee; but needs 2 currencies). For neither Dex nor MetaDex can it (offers.dat) be used to re-create the history of TXs with full details (that's what MPtxlist + blochain + block-explorer-like-entity is for). The "subaction" command (modifier) is to be executed in the TX/block it was received -- it modifies the orderbook (Trades, Adds, Subtracts, Cancels, etc.) right then & there. offers.dat store the state of the orderbook on disk, during shutdown -- by then any individual subaction's meaning is long out of the picture. Hope this is not too confusing. :) P.S. but yes we have extra fields in offers_.dat today which can be overloaded just like Faiz did. I ony want to say I do not believe storing the "subaction" is meaningful or useful.

[dexX7]

Basically you could reconstruct state by decoding all transactions every time - or store everything and anything with all information. This sounds as you intend something inbetween and store the minimum amount of data that is sufficient to reconstruct the latest state, but not sufficient to reconstruct every data that is created during the whole process. Is this correct?

MP_txlist + blochain + block-explorer-like-entity

This means such an entity would be required to parse and interpret all transactions correctly - without #143 this could result in a consensus hell (mismatch) which might not be noticed for a longer time.

[m21]

For this subsystem (trading, Dex, Metadex) my goal is to ensure I have enough information to perform trades and to Add, Cancel, Subtract new commands as they come in -- correctly. The persistence patch helps ensure that the information is still available during the client shutdown/restart/reorg. I am guaranteeing the client works in the present, starting at some block (Exodus or later) -- parsing forward. That's about all we can do here, for backward-parsing tricks & historical reconstruction & continuing in https://github.com/mastercoin-MSC/mastercore/issues/143 .

[m21]

Sorry for the close, fat finger. Re-opened.

[ghost]

The "subaction" command (modifier) is to be executed in the TX/block it was received -- 
it modifies the orderbook (Trades, Adds, Subtracts, Cancels, etc.) right then & there. 
offers*.dat store the state of the orderbook on disk, during shutdown -- by then any
individual subaction's meaning is long out of the picture.

Yes I understand the meaning here, perhaps this is something to display/store for historical transactions and not to be shown for the orderbook? If I place an order and want to see it in gettransaction_MP for example, I would still like to know the historical value of that field at the time I placed the trade, or is there an argument for that not being so valuable?

Also, I looked at the MP_txlist dump and I think offers are already stored there, @m21 can you verify that nothing needs to be done to include metadex transactions in MP_txlist? Or was prior discussion centering around making other values available using MP_txlist (?) , I would really appreciate some elaboration on this.

[m21]

Good questions on MP_txlist Faiz. We should extend the entry of data and make retrieval function(s) more flexible, based on the TX Type & subaction (if any). Also - please refactor the code so I can merge it. Thanks

[ghost]

Closing in favor of split PR