Store PMMR file position index to mitigate data corruption

[yeastplume]

Augment PMMR storage data to also contain a metadata file that contains the byte-sizes of the PMMR as of the last N blocks. This may have several benefits, but is mostly to aid recovery in the event of corruption

• Metadata file will be written to as the absolute last action when committing a block.
• On start up and PMMR loading, the system can truncate to the last written size. If that results in invalid data, system can attempt to read earlier indices to find a block/position pairing that results in valid data.

[yeastplume]

Will address https://github.com/mimblewimble/grin/issues/371

[ignopeverell]

Just a thought. The chain head is already written last and is what the chain, on restart, uses as a basis to know where it's at. So maybe add the PMMR sizes/positions to that would make sense. The downside is that there's only one head, so it wouldn't work to store it for the last N blocks. But the head has the latest block hash as well as the one before last.

[antiochp]

That might be a nice simple solution and would avoid another file to manage. Do we trust rocksdb in this scenario? I assume we do, we don't see issues with writes to the db, the issue is non-atomicity between writing the pmmr file and the associated rocksdb indices.

[ignopeverell]

Right, it would be more a 2-phase-commit, XA type of thing. And we don't really care about either of those things as long as we get to write one last authoritative record at the end.

[yeastplume]

Yes, have just been looking at it and storing it in rocksdb seems to be far simpler and makes sense. We can start off storing positions for the last block and last block-1, and if that turns out to be inadequate we can look at further options.

Will implement this and then try and create a test or two that corrupts data and recover.

[yeastplume]

Done via #739