Open markrmiller opened 3 weeks ago
Thanks for reporting. In my environment, it takes about 10 seconds for 3,000 images with the current implementation of sd-scripts. Do you know what causes the difference, between 20 minutes and 10 seconds?
do you have a fast ssd?
The data is on the HDD.
INFO get image size from name of cache files train_util.py:1745
100%|███████████████████████████████████████████████████████████████████████████████| 906/906 [00:03<00:00, 254.30it/s]
2024-08-22 12:11:45 INFO set image size from cache files: 906/906 train_util.py:1752
INFO found directory xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx train_util.py:1754
contains 906 image files
2024-08-22 12:11:48 INFO get image size from name of cache files train_util.py:1745
100%|██████████████████████████████████████████████████████████████████████████████| 136/136 [00:00<00:00, 1606.09it/s]
INFO set image size from cache files: 136/136 train_util.py:1752
INFO found directory xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx train_util.py:1754
contains 136 image files
INFO get image size from name of cache files train_util.py:1745
100%|███████████████████████████████████████████████████████████████████████████████| 180/180 [00:00<00:00, 767.91it/s]
INFO set image size from cache files: 180/180 train_util.py:1752
INFO found directory xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx train_util.py:1754
contains 180 image files
2024-08-22 12:11:49 INFO get image size from name of cache files train_util.py:1745
100%|███████████████████████████████████████████████████████████████████████████████| 180/180 [00:00<00:00, 822.88it/s]
and 4 more directories.
Thanks for reporting. It's very strange, but the following line may take an extremely long time on certain systems:
npz_file = glob.glob(os.path.splitext(absolute_path)[0] + "_*" + FluxLatentsCachingStrategy.FLUX_LATENTS_NPZ_SUFFIX)
I will try changing the processing to avoid using globs as much as possible.
I added a new branch fast_image_sizes
. Could you please test with the branch?
The new branch fixed this issue for me.
When I train with 4000 images, it takes forever to start training because just collecting the image sizes from the npz filenames takes 20 minutes from the drive where I have the data.
This code takes...I didn't measure, but maybe 10 seconds?