search

streamingfast / merger

Apache License 2.0
4 stars 5 forks source link

When merger skips files, the seen-blocks cache mechanism makes it put all one-blocks-files in the next merged #1

Closed sduchesneau closed 4 years ago

sduchesneau commented 4 years ago

scenario:

  1. one-blocks-files exist from 100 to 500, but missing specific file 125 (block hole)
  2. Someone restarts a mindreader from snapshot, creating merged files 100, 200, 300, 400, 500, then going back to creating 501, 502, etc. as one-block files. (this is now default behavior for mindreader on develop, when the blocks are old enough or clearly behind LIB, to prevent creating gazillions of one-block files when we know there won't be any fork...)
  3. The merger (current behavior) sees the merged files produced (100, 200, 300...) so it skips these bundles, and starts merging from 500, 501, etc., deleting the one-block-files behind.

Issues:

  1. The merger does NOT put these one-block-files in the seen-blocks cache.
  2. Deletion batches have a maximum size. a request to delete too many one-block-files will ignore them passed a certain point and reevaluate next time. (deletion could also fail if the merger is restarted at that point)
  3. When those remaining one-block-files are evaluated, since they are not older than max-fixable-fork (~ usually +5000 blocks), and they are not in seen-blocks-cache, they are considered VALID (remember that they were not merged by the merger itself, but by the mindreader in automatic catch up mode ..). So they will all be put in the next merged blocks (it may end up with a few thousand blocks !!)
  4. The one-block-files that correspond to forked blocks will be skipped and deleted if the merger just takes the merged files from mindreader catching up. This results in possibly customers having been fed blocks, but those blocks are unavailable later (invalid cursors, etc.), and breaks the guarantee to always be able to resolve forks later on.

Solution to be implemented:

  1. The merger should keep the last written block bundle on local disk (in its 'progress' file) and, if it exists, restart from there. It should only "try to find starting block" once...

    • change doc around --merger-progress-filename, it will now be used in live mode too!
    • maybe tweak flags around batch mode to be more consistent with other apps

  2. The merger will try to create bundle 100->199 from one-block-files. If it cannot, AND it sees the merged-blocks.100 created by someone else, it will download it, list the existing blocks and populate its seen-cache.

  3. The merger will then look at 200-299 range, try to create it itself. if it cannot, look for existing merged-block, again, reading it to insert its own blocks and overwrite...

NOTE: we cannot modify already published merged-blocks-files... so....

sduchesneau commented 4 years ago

either we have mindreader publish merged-blocks-file inside the temporary location (currently called one-block-files) and the merger can use it (careful on resource usages, then :( )

or we keep pushing the blocks that were not in the "seen" cache further and further away (that's the chosen scenario)