Closed Maw-Fox closed 1 year ago
Contemplating closing this pull request as I'm creating a branch to both optimize performance and allow dataset migration and this is relevant to such.
--resize
is True
, --data_migration
(default: True
), data migration would create a clone of the relative path to the dataset path, and output %DATASET%_cropped
adjacent to the dataset directory. This is to make sure this expensive operation is only run once..load()
operation substantially extends validation time (from ~1000f/s to ~100f/s), while being more extensive it feels like for already-prepared datasets the slowdown is not worth it, despite the lack of verbose debugging upon hitting truncated/corrupted files. Subsequently the addition of --extended_validation
(default: False
) would allow users to easily perform one-time extended validation.--resize
default migration, further speeding up operations on already-validated datasets.
Fixes
convert()
operations, as it stands the error that is uncaught when this runs does not inform the user which file produced the error, this does that as well as skips the file in question.