streamingfast / firehose-core

Firehose Integrators Tool Kit (for `firehose-<chain>` maintainers)
Apache License 2.0
11 stars 10 forks source link

Add configuration parameter merger-one-block-deletion-threads #7

Closed matthewdarwin closed 1 year ago

matthewdarwin commented 1 year ago

In old dfuse, there was a configuration parameter merger-one-block-deletion-threads. This doesn't exist in firehose.

Request to re-add it as antelope firehose generates a lot of one-block files and currently takes way too long for merger to delete them when one of the readers starts from an older snapshot.

We have merger-one-block-deletion-threads: 25 set in dfuse for example.

matthewdarwin commented 1 year ago

Actually this is already tracked: https://github.com/streamingfast/merger/issues/24