I have mindreader producing lots of one-block files, catching up from block 1.
$ ls mindreader/work/uploadable-oneblock | wc -l
2601403
It can produce files faster they can be uploaded to s3 bucket.
In the older merger, there was a hash structure here to sort the one-block files into directories of 1000 at a time or so. This is important on ext4 filesystem where performance of listing files on large directories is poor.
matthewdarwin
commented
2 years ago
I have mindreader producing lots of one-block files, catching up from block 1.
It can produce files faster they can be uploaded to s3 bucket.
In the older merger, there was a hash structure here to sort the one-block files into directories of 1000 at a time or so. This is important on ext4 filesystem where performance of listing files on large directories is poor.