newer mindreader needs to have a hash-structure under uploadable-oneblock

[matthewdarwin]

I have mindreader producing lots of one-block files, catching up from block 1.

$ ls mindreader/work/uploadable-oneblock | wc -l
2601403

It can produce files faster they can be uploaded to s3 bucket.

In the older merger, there was a hash structure here to sort the one-block files into directories of 1000 at a time or so. This is important on ext4 filesystem where performance of listing files on large directories is poor.

[matthewdarwin]

Maybe this is less of an issue after the cleanup of one-block files was implemented.

[sduchesneau]

It should not be a big issue since now: 1) the direct merging from node-manager is fixed (no more holes are produced when stopped/restarted...) 2) the file deletion is now properly implemented

You should not see situations with 2.6M files appear again.

Let me know if you still see this issue come up or if direct merging (--mindreader-node-merge-and-store-directly) is still problematic for you after the recent fixes.

[matthewdarwin]

There still could be a problem if s3 is unavailable for some reason and we're trying to upload one-blocks and the blocks all pile up. We've had this happen before (s3 being unavailable).

[matthewdarwin]

But other than this weird case, then I agree the problem is much less now.

[matthewdarwin]

lots of one-block files piled up in mindreader/work/mergable due to bug https://github.com/streamingfast/node-manager/issues/29 So this issue is still applicable in error cases.

[maoueh]

Closing, every thing is found under https://github.com/streamingfast/firehose-core now anyway. Going to make this repo read-only.