When an image dataset is imbalanced, where a subset of clustered (in time) images takes much longer to process than the rest, the block chunking we are using now may result in very imbalanced chunks, and the processing time dominated by one of the chunks. Adding striping or other strategies for splitting up the file list may yield better results in these cases.
When an image dataset is imbalanced, where a subset of clustered (in time) images takes much longer to process than the rest, the block chunking we are using now may result in very imbalanced chunks, and the processing time dominated by one of the chunks. Adding striping or other strategies for splitting up the file list may yield better results in these cases.