Ceph (when running as s3 server) is known to be very slow running deletes. Add configuration to merger to specify number of parallel delete threads as the out-of-box default (8) is not sufficient to keep up when doing initial processing of blocks. May need 100 threads.
In old dfuse, there was a configuration parameter merger-one-block-deletion-threads. This doesn't exist in firehose.
Request to re-add it as antelope firehose generates a lot of one-block files and currently takes way too long for merger to delete them when one of the readers starts from an older snapshot.
We have merger-one-block-deletion-threads: 25 set in dfuse for example.
Ceph (when running as s3 server) is known to be very slow running deletes. Add configuration to merger to specify number of parallel delete threads as the out-of-box default (8) is not sufficient to keep up when doing initial processing of blocks. May need 100 threads.
In old dfuse, there was a configuration parameter
merger-one-block-deletion-threads. This doesn't exist in firehose.Request to re-add it as antelope firehose generates a lot of
one-blockfiles and currently takes way too long for merger to delete them when one of the readers starts from an older snapshot.We have merger-one-block-deletion-threads: 25 set in dfuse for example.