search

streamingfast / node-manager

Wrapping process to operate blockchain nodes.
https://dfuse.io
Apache License 2.0
7 stars 9 forks source link

Enhancement: only upload blocks if merger needs them #21

Closed matthewdarwin closed 4 months ago

matthewdarwin commented 2 years ago

If I set mindreader-node-merge-threshold-block-age, then mindreader will sometimes merge blocks and sometimes not. The location to upload files to is either: common-oneblock-store-url or common-blocks-store-url.

It would be more efficient to skip merging and uploading entirely if the merger doesn't need them.

Currently mindreader can sync with blockmeta to understand where the chain is at, but it doesn't understand where the merger is at. If there was a way mindreader could understand merger's status then it could avoid sending blocks that are not required.

sduchesneau commented 2 years ago

The blockmeta (with its database, etc.) is not part of the "modern, lean" version of firehose, so we don't want to depend on it or build more features around it at the time.

The merger is able to easily skip/delete blocks that are too old, or to ignore blocks that are way ahead. I'm not sure what you are trying to "save" by skipping the upload of merged files...

matthewdarwin commented 2 years ago

I'm only trying to save processing load. If you have an infrastructure that is just booting up (after a crash or whatever), then things are very overloaded as things get in sync. The more 'unecessary' processing load that can be removed, the better (speaking from an operations point of view).

This is low priority.

matthewdarwin commented 2 years ago

Just to be clear, we didn't deploy blockmeta service for SF (only there for older dfuse), so not asking for it to come back. ;-)

maoueh commented 4 months ago

Closing, every thing is found under https://github.com/streamingfast/firehose-core now anyway. Going to make this repo read-only.