search

n00mkrad / text2image-gui

Somewhat modular text2image GUI, initially just for Stable Diffusion
GNU General Public License v3.0
921 stars 93 forks source link

RAM memory hogging and weird model "scanning" behaviour. #70

Open mart-hill opened 1 year ago

mart-hill commented 1 year ago

I noticed that with trying Dreambooth training in version 1.9.0, I'm unable to do it due to hogging the 64GB of RAM and 16GB of swap, and it's on the "poorest" quality setting. The training goes only on first 5 steps, then it stops and tries to do something (presumably save the last.ckpt file?), but it hogs the memory at that point and doesn't finish. My VRAM and RAM is fine, I use RTX 3090 and no background apps. I have to mention, that the base model is from the second folder, not from the default path. Also, I noticed, that generating images from such "second folder" model doesn't work, cause the GUI tries to read every model file until it "meets" the one I've chosen. I observed that behaviour on Process Hacker/System Informer. The image generation never happens, cause that "scanning" process takes a LOT of time (it goes alphabetically, I think). The second path for my models is on SSD, so speed isn't the issue, disk space neither (during saving last.ckpt file).

I also noticed, that InstructPix2Pix won't generate any image, despite having all the models downloaded, unless I allow ip2p_batch.py to connect to the internet to huggingface.co server (54.235.118.239 and 3.231.67.228 IP addresses), why?

n00mkrad commented 1 year ago

InstructPix2Pix not working offline has been fixed. Will look into Dreambooth now

mart-hill commented 1 year ago
image

That's how the memory usage looks like during my attempt to train with (now v1.9.1) the app.


Training on 200 images to 4000 steps using learning rate of 3.6E-06.
Starting training on GPU 0.
Log Folder: Data\sessions\2023-02-03-08-40-25\db\1675414053700
Training (Step 6/4000 - 0% - ETA: 02:51:04)...
Training failed - model file was not saved.

in the dreambooth log:

[00000165] [02-03-2023 08:49:01] Project config
[00000166] [02-03-2023 08:49:01] model:
[00000167] [02-03-2023 08:49:01]   base_learning_rate: 3.6e-06
[00000168] [02-03-2023 08:49:01]   target: ldm.models.diffusion.ddpm.LatentDiffusion
[00000169] [02-03-2023 08:49:01]   params:
[00000170] [02-03-2023 08:49:01]     reg_weight: 1.0
[00000171] [02-03-2023 08:49:01]     linear_start: 0.00085
[00000172] [02-03-2023 08:49:01]     linear_end: 0.012
[00000173] [02-03-2023 08:49:01]     num_timesteps_cond: 1
[00000174] [02-03-2023 08:49:01]     log_every_t: 200
[00000175] [02-03-2023 08:49:01]     timesteps: 1000
[00000176] [02-03-2023 08:49:01]     first_stage_key: image
[00000177] [02-03-2023 08:49:01]     cond_stage_key: caption
[00000178] [02-03-2023 08:49:01]     image_size: 64
[00000179] [02-03-2023 08:49:01]     channels: 4
[00000180] [02-03-2023 08:49:01]     cond_stage_trainable: true
[00000181] [02-03-2023 08:49:01]     conditioning_key: crossattn
[00000182] [02-03-2023 08:49:01]     monitor: val/loss_simple_ema
[00000183] [02-03-2023 08:49:01]     scale_factor: 0.18215
[00000184] [02-03-2023 08:49:01]     use_ema: false
[00000185] [02-03-2023 08:49:01]     embedding_reg_weight: 0.0
[00000186] [02-03-2023 08:49:01]     unfreeze_model: true
[00000187] [02-03-2023 08:49:01]     model_lr: 1.0e-06
[00000188] [02-03-2023 08:49:01]     personalization_config:
[00000189] [02-03-2023 08:49:01]       target: ldm.modules.embedding_manager.EmbeddingManager
[00000190] [02-03-2023 08:49:01]       params:
[00000191] [02-03-2023 08:49:01]         placeholder_strings:
[00000192] [02-03-2023 08:49:01]         - '*'
[00000193] [02-03-2023 08:49:01]         initializer_words:
[00000194] [02-03-2023 08:49:01]         - sculpture
[00000195] [02-03-2023 08:49:01]         per_image_tokens: false
[00000196] [02-03-2023 08:49:01]         num_vectors_per_token: 1
[00000197] [02-03-2023 08:49:01]         progressive_words: false
[00000198] [02-03-2023 08:49:01]     unet_config:
[00000199] [02-03-2023 08:49:01]       target: ldm.modules.diffusionmodules.openaimodel.UNetModel
[00000200] [02-03-2023 08:49:01]       params:
[00000201] [02-03-2023 08:49:01]         image_size: 32
[00000202] [02-03-2023 08:49:01]         in_channels: 4
[00000203] [02-03-2023 08:49:01]         out_channels: 4
[00000204] [02-03-2023 08:49:01]         model_channels: 320
[00000205] [02-03-2023 08:49:01]         attention_resolutions:
[00000206] [02-03-2023 08:49:01]         - 4
[00000207] [02-03-2023 08:49:01]         - 2
[00000208] [02-03-2023 08:49:01]         - 1
[00000209] [02-03-2023 08:49:01]         num_res_blocks: 2
[00000210] [02-03-2023 08:49:01]         channel_mult:
[00000211] [02-03-2023 08:49:01]         - 1
[00000212] [02-03-2023 08:49:01]         - 2
[00000213] [02-03-2023 08:49:01]         - 4
[00000214] [02-03-2023 08:49:01]         - 4
[00000215] [02-03-2023 08:49:01]         num_heads: 8
[00000216] [02-03-2023 08:49:01]         use_spatial_transformer: true
[00000217] [02-03-2023 08:49:01]         transformer_depth: 1
[00000218] [02-03-2023 08:49:01]         context_dim: 768
[00000219] [02-03-2023 08:49:01]         use_checkpoint: true
[00000220] [02-03-2023 08:49:01]         legacy: false
[00000221] [02-03-2023 08:49:01]     first_stage_config:
[00000222] [02-03-2023 08:49:01]       target: ldm.models.autoencoder.AutoencoderKL
[00000223] [02-03-2023 08:49:01]       params:
[00000224] [02-03-2023 08:49:01]         embed_dim: 4
[00000225] [02-03-2023 08:49:01]         monitor: val/rec_loss
[00000226] [02-03-2023 08:49:01]         ddconfig:
[00000227] [02-03-2023 08:49:01]           double_z: true
[00000228] [02-03-2023 08:49:01]           z_channels: 4
[00000229] [02-03-2023 08:49:01]           resolution: 512
[00000230] [02-03-2023 08:49:01]           in_channels: 3
[00000231] [02-03-2023 08:49:01]           out_ch: 3
[00000232] [02-03-2023 08:49:01]           ch: 128
[00000233] [02-03-2023 08:49:01]           ch_mult:
[00000234] [02-03-2023 08:49:01]           - 1
[00000235] [02-03-2023 08:49:01]           - 2
[00000236] [02-03-2023 08:49:01]           - 4
[00000237] [02-03-2023 08:49:01]           - 4
[00000238] [02-03-2023 08:49:01]           num_res_blocks: 2
[00000239] [02-03-2023 08:49:01]           attn_resolutions: []
[00000240] [02-03-2023 08:49:01]           dropout: 0.0
[00000241] [02-03-2023 08:49:01]         lossconfig:
[00000242] [02-03-2023 08:49:01]           target: torch.nn.Identity
[00000243] [02-03-2023 08:49:01]     cond_stage_config:
[00000244] [02-03-2023 08:49:01]       target: ldm.modules.encoders.modules.FrozenCLIPEmbedder
[00000245] [02-03-2023 08:49:01]     ckpt_path: O:/AI/stable-diffusion-webui/models/Stable-diffusion/sd_v1-5_vae.ckpt
[00000246] [02-03-2023 08:49:01] data:
[00000247] [02-03-2023 08:49:01]   target: main.DataModuleFromConfig
[00000248] [02-03-2023 08:49:01]   params:
[00000249] [02-03-2023 08:49:01]     batch_size: 1
[00000250] [02-03-2023 08:49:01]     num_workers: 1
[00000251] [02-03-2023 08:49:01]     wrap: false
[00000252] [02-03-2023 08:49:01]     train:
[00000253] [02-03-2023 08:49:01]       target: ldm.data.personalized.PersonalizedBase
[00000254] [02-03-2023 08:49:01]       params:
[00000255] [02-03-2023 08:49:01]         size: 512
[00000256] [02-03-2023 08:49:01]         set: train
[00000257] [02-03-2023 08:49:01]         per_image_tokens: false
[00000258] [02-03-2023 08:49:01]         repeats: 100
[00000259] [02-03-2023 08:49:01]         placeholder_token: persom
[00000260] [02-03-2023 08:49:01]     reg:
[00000261] [02-03-2023 08:49:01]       target: ldm.data.personalized.PersonalizedBase
[00000262] [02-03-2023 08:49:01]       params:
[00000263] [02-03-2023 08:49:01]         size: 512
[00000264] [02-03-2023 08:49:01]         set: train
[00000265] [02-03-2023 08:49:01]         reg: true
[00000266] [02-03-2023 08:49:01]         per_image_tokens: false
[00000267] [02-03-2023 08:49:01]         repeats: 100
[00000268] [02-03-2023 08:49:01]         placeholder_token: person
[00000269] [02-03-2023 08:49:01]     validation:
[00000270] [02-03-2023 08:49:01]       target: ldm.data.personalized.PersonalizedBase
[00000271] [02-03-2023 08:49:01]       params:
[00000272] [02-03-2023 08:49:01]         size: 512
[00000273] [02-03-2023 08:49:01]         set: val
[00000274] [02-03-2023 08:49:01]         per_image_tokens: false
[00000275] [02-03-2023 08:49:01]         repeats: 10
[00000276] [02-03-2023 08:49:01]         placeholder_token: person
[00000277] [02-03-2023 08:49:01] Lightning config
[00000278] [02-03-2023 08:49:01] modelcheckpoint:
[00000279] [02-03-2023 08:49:01]   params:
[00000280] [02-03-2023 08:49:01]     every_n_train_steps: 4001
[00000281] [02-03-2023 08:49:01] callbacks:
[00000282] [02-03-2023 08:49:01]   image_logger:
[00000283] [02-03-2023 08:49:01]     target: main.ImageLogger
[00000284] [02-03-2023 08:49:01]     params:
[00000285] [02-03-2023 08:49:01]       batch_frequency: 1000
[00000286] [02-03-2023 08:49:01]       max_images: 8
[00000287] [02-03-2023 08:49:01]       increase_log_steps: false
[00000288] [02-03-2023 08:49:01] trainer:
[00000289] [02-03-2023 08:49:01]   benchmark: true
[00000290] [02-03-2023 08:49:01]   max_steps: 4000
[00000291] [02-03-2023 08:49:01]   gpus: 0,
[00000292] [02-03-2023 08:49:07] X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py:240: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 28 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
[00000293] [02-03-2023 08:49:07]   rank_zero_warn(
[00000294] [02-03-2023 08:49:11] Sanity Checking: 0it [00:00, ?it/s]
[00000295] [02-03-2023 08:49:11] Sanity Checking:   0%|          | 0/2 [00:00<?, ?it/s]
[00000296] [02-03-2023 08:49:29] Sanity Checking DataLoader 0:   0%|          | 0/2 [00:00<?, ?it/s]
[00000297] [02-03-2023 08:49:29] Sanity Checking DataLoader 0:  50%|#####     | 1/2 [00:17<00:17, 17.19s/it]
[00000298] [02-03-2023 08:49:29] Sanity Checking DataLoader 0:  50%|#####     | 1/2 [00:17<00:17, 17.19s/it]
[00000299] [02-03-2023 08:49:29] Sanity Checking DataLoader 0: 100%|##########| 2/2 [00:17<00:00,  7.29s/it]
[00000300] [02-03-2023 08:49:29] Sanity Checking DataLoader 0: 100%|##########| 2/2 [00:17<00:00,  7.29s/it]
[00000301] [02-03-2023 08:49:29] X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py:240: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 28 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
[00000302] [02-03-2023 08:49:29]   rank_zero_warn(
[00000303] [02-03-2023 08:49:29] X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py:2102: LightningDeprecationWarning: `Trainer.root_gpu` is deprecated in v1.6 and will be removed in v1.8. Please use `Trainer.strategy.root_device.index` instead.
[00000304] [02-03-2023 08:49:29]   rank_zero_deprecation(
[00000305] [02-03-2023 08:49:29] X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py:2102: LightningDeprecationWarning: `Trainer.root_gpu` is deprecated in v1.6 and will be removed in v1.8. Please use `Trainer.strategy.root_device.index` instead.
[00000306] [02-03-2023 08:49:29]   rank_zero_deprecation(
[00000307] [02-03-2023 08:49:29] Training: 0it [00:00, ?it/s]
[00000309] [02-03-2023 08:49:29] Training:   0%|          | 0/20200 [00:00<?, ?it/s]
[00000311] [02-03-2023 08:49:33] X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\utilities\data.py:72: UserWarning: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 1. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`.
[00000312] [02-03-2023 08:49:33]   warning_cache.warn(
[00000313] [02-03-2023 08:49:33] X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\connectors\logger_connector\result.py:229: UserWarning: You called `self.log('global_step', ...)` in your `training_step` but the value needs to be floating point. Converting it to torch.float32.
[00000314] [02-03-2023 08:49:33]   warning_cache.warn(
[00000315] [02-03-2023 08:49:38] Epoch 0:   0%|          | 0/20200 [00:00<?, ?it/s] 
[00000316] [02-03-2023 08:49:38] Epoch 0:   0%|          | 1/20200 [00:08<47:44:33,  8.51s/it]
[00000317] [02-03-2023 08:49:38] Epoch 0:   0%|          | 1/20200 [00:08<47:44:33,  8.51s/it]
[00000318] [02-03-2023 08:49:39] Epoch 0:   0%|          | 1/20200 [00:08<47:59:42,  8.55s/it, loss=0.036, v_num=0, train/loss_simple_step=0.00438, train/loss_vlb_step=2.52e-5, train/loss_step=0.00438, global_step=0.000]
[00000320] [02-03-2023 08:49:39] Epoch 0:   0%|          | 2/20200 [00:10<28:10:50,  5.02s/it, loss=0.036, v_num=0, train/loss_simple_step=0.00438, train/loss_vlb_step=2.52e-5, train/loss_step=0.00438, global_step=0.000]
[00000322] [02-03-2023 08:49:39] Epoch 0:   0%|          | 2/20200 [00:10<28:10:50,  5.02s/it, loss=0.036, v_num=0, train/loss_simple_step=0.00438, train/loss_vlb_step=2.52e-5, train/loss_step=0.00438, global_step=0.000]
[00000324] [02-03-2023 08:49:41] Epoch 0:   0%|          | 2/20200 [00:10<28:28:35,  5.08s/it, loss=0.15, v_num=0, train/loss_simple_step=0.0366, train/loss_vlb_step=0.000136, train/loss_step=0.0366, global_step=1.000]  
[00000326] [02-03-2023 08:49:41] Epoch 0:   0%|          | 3/20200 [00:11<21:51:46,  3.90s/it, loss=0.15, v_num=0, train/loss_simple_step=0.0366, train/loss_vlb_step=0.000136, train/loss_step=0.0366, global_step=1.000]
[00000328] [02-03-2023 08:49:41] Epoch 0:   0%|          | 3/20200 [00:11<21:51:46,  3.90s/it, loss=0.15, v_num=0, train/loss_simple_step=0.0366, train/loss_vlb_step=0.000136, train/loss_step=0.0366, global_step=1.000]
[00000330] [02-03-2023 08:49:42] Epoch 0:   0%|          | 3/20200 [00:11<21:58:43,  3.92s/it, loss=0.305, v_num=0, train/loss_simple_step=0.0412, train/loss_vlb_step=0.000148, train/loss_step=0.0412, global_step=2.000]
[00000332] [02-03-2023 08:49:42] Epoch 0:   0%|          | 4/20200 [00:13<18:30:07,  3.30s/it, loss=0.305, v_num=0, train/loss_simple_step=0.0412, train/loss_vlb_step=0.000148, train/loss_step=0.0412, global_step=2.000]
[00000334] [02-03-2023 08:49:42] Epoch 0:   0%|          | 4/20200 [00:13<18:30:10,  3.30s/it, loss=0.305, v_num=0, train/loss_simple_step=0.0412, train/loss_vlb_step=0.000148, train/loss_step=0.0412, global_step=2.000]
[00000336] [02-03-2023 08:49:44] Epoch 0:   0%|          | 4/20200 [00:13<18:49:16,  3.35s/it, loss=0.311, v_num=0, train/loss_simple_step=0.294, train/loss_vlb_step=0.00249, train/loss_step=0.294, global_step=3.000]   
[00000338] [02-03-2023 08:49:44] Epoch 0:   0%|          | 5/20200 [00:14<16:24:42,  2.93s/it, loss=0.311, v_num=0, train/loss_simple_step=0.294, train/loss_vlb_step=0.00249, train/loss_step=0.294, global_step=3.000]
[00000340] [02-03-2023 08:49:44] Epoch 0:   0%|          | 5/20200 [00:14<16:24:42,  2.93s/it, loss=0.311, v_num=0, train/loss_simple_step=0.294, train/loss_vlb_step=0.00249, train/loss_step=0.294, global_step=3.000]
[00000342] [02-03-2023 08:49:45] Epoch 0:   0%|          | 5/20200 [00:14<16:28:24,  2.94s/it, loss=0.304, v_num=0, train/loss_simple_step=0.0939, train/loss_vlb_step=0.000316, train/loss_step=0.0939, global_step=4.000]
[00000344] [02-03-2023 08:49:45] Epoch 0:   0%|          | 6/20200 [00:15<14:52:07,  2.65s/it, loss=0.304, v_num=0, train/loss_simple_step=0.0939, train/loss_vlb_step=0.000316, train/loss_step=0.0939, global_step=4.000]
[00000346] [02-03-2023 08:49:45] Epoch 0:   0%|          | 6/20200 [00:15<14:52:07,  2.65s/it, loss=0.304, v_num=0, train/loss_simple_step=0.0939, train/loss_vlb_step=0.000316, train/loss_step=0.0939, global_step=4.000]
[00000348] [02-03-2023 08:49:47] Epoch 0:   0%|          | 6/20200 [00:16<15:06:42,  2.69s/it, loss=0.316, v_num=0, train/loss_simple_step=0.00648, train/loss_vlb_step=3.52e-5, train/loss_step=0.00648, global_step=5.000]
[00000350] [02-03-2023 08:49:47] Epoch 0:   0%|          | 7/20200 [00:17<14:14:37,  2.54s/it, loss=0.316, v_num=0, train/loss_simple_step=0.00648, train/loss_vlb_step=3.52e-5, train/loss_step=0.00648, global_step=5.000]
[00000352] [02-03-2023 08:49:47] Epoch 0:   0%|          | 7/20200 [00:17<14:14:37,  2.54s/it, loss=0.316, v_num=0, train/loss_simple_step=0.00648, train/loss_vlb_step=3.52e-5, train/loss_step=0.00648, global_step=5.000]
[00000354] [02-03-2023 08:52:02] Epoch 0:   0%|          | 7/20200 [00:17<14:24:51,  2.57s/it, loss=0.293, v_num=0, train/loss_simple_step=0.0913, train/loss_vlb_step=0.0003, train/loss_step=0.0913, global_step=6.000]   Summoning checkpoint.
[00000355] [02-03-2023 08:52:02] Traceback (most recent call last):
[00000357] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\repo\db\main.py", line 837, in <module>
[00000358] [02-03-2023 08:52:02]     trainer.fit(model, data)
[00000359] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 770, in fit
[00000360] [02-03-2023 08:52:02]     self._call_and_handle_interrupt(
[00000361] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 723, in _call_and_handle_interrupt
[00000362] [02-03-2023 08:52:02]     return trainer_fn(*args, **kwargs)
[00000363] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 811, in _fit_impl
[00000364] [02-03-2023 08:52:02]     results = self._run(model, ckpt_path=self.ckpt_path)
[00000365] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1236, in _run
[00000366] [02-03-2023 08:52:02]     results = self._run_stage()
[00000367] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1323, in _run_stage
[00000368] [02-03-2023 08:52:02]     return self._run_train()
[00000369] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1353, in _run_train
[00000370] [02-03-2023 08:52:02]     self.fit_loop.run()
[00000371] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\loops\base.py", line 204, in run
[00000372] [02-03-2023 08:52:02]     self.advance(*args, **kwargs)
[00000373] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 266, in advance
[00000374] [02-03-2023 08:52:02]     self._outputs = self.epoch_loop.run(self._data_fetcher)
[00000375] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\loops\base.py", line 204, in run
[00000376] [02-03-2023 08:52:02]     self.advance(*args, **kwargs)
[00000377] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 171, in advance
[00000378] [02-03-2023 08:52:02]     batch = next(data_fetcher)
[00000379] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 184, in __next__
[00000380] [02-03-2023 08:52:02]     return self.fetching_function()
[00000381] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 259, in fetching_function
[00000382] [02-03-2023 08:52:02]     self._fetch_next_batch(self.dataloader_iter)
[00000383] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 273, in _fetch_next_batch
[00000384] [02-03-2023 08:52:02]     batch = next(iterator)
[00000385] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 558, in __next__
[00000386] [02-03-2023 08:52:02]     return self.request_next_batch(self.loader_iters)
[00000387] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 570, in request_next_batch
[00000388] [02-03-2023 08:52:02]     return apply_to_collection(loader_iters, Iterator, next)
[00000389] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\pytorch_lightning\utilities\apply_func.py", line 99, in apply_to_collection
[00000390] [02-03-2023 08:52:02]     return function(data, *args, **kwargs)
[00000391] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\torch\utils\data\dataloader.py", line 530, in __next__
[00000392] [02-03-2023 08:52:02]     data = self._next_data()
[00000393] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\torch\utils\data\dataloader.py", line 1224, in _next_data
[00000394] [02-03-2023 08:52:02]     return self._process_data(data)
[00000395] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\torch\utils\data\dataloader.py", line 1250, in _process_data
[00000396] [02-03-2023 08:52:02]     data.reraise()
[00000397] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\torch\_utils.py", line 457, in reraise
[00000398] [02-03-2023 08:52:02]     raise exception
[00000399] [02-03-2023 08:52:02] ValueError: Caught ValueError in DataLoader worker process 0.
[00000400] [02-03-2023 08:52:02] Original Traceback (most recent call last):
[00000401] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\torch\utils\data\_utils\worker.py", line 287, in _worker_loop
[00000402] [02-03-2023 08:52:02]     data = fetcher.fetch(index)
[00000403] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\torch\utils\data\_utils\fetch.py", line 49, in fetch
[00000404] [02-03-2023 08:52:02]     data = [self.dataset[idx] for idx in possibly_batched_index]
[00000405] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\torch\utils\data\_utils\fetch.py", line 49, in <listcomp>
[00000406] [02-03-2023 08:52:02]     data = [self.dataset[idx] for idx in possibly_batched_index]
[00000407] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\repo\db\main.py", line 222, in __getitem__
[00000408] [02-03-2023 08:52:02]     return tuple(d[idx] for d in self.datasets)
[00000409] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\repo\db\main.py", line 222, in <genexpr>
[00000410] [02-03-2023 08:52:02]     return tuple(d[idx] for d in self.datasets)
[00000411] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\repo\db\ldm\data\personalized.py", line 188, in __getitem__
[00000412] [02-03-2023 08:52:02]     image = Image.open(self.image_paths[i % self.num_images])
[00000413] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\PIL\Image.py", line 3133, in open
[00000414] [02-03-2023 08:52:02]     im = _open_core(fp, filename, prefix, formats)
[00000415] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\PIL\Image.py", line 3119, in _open_core
[00000416] [02-03-2023 08:52:02]     im = factory(fp, filename)
[00000417] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\PIL\ImageFile.py", line 116, in __init__
[00000418] [02-03-2023 08:52:02]     self._open()
[00000419] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\PIL\PngImagePlugin.py", line 730, in _open
[00000420] [02-03-2023 08:52:02]     s = self.png.call(cid, pos, length)
[00000421] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\PIL\PngImagePlugin.py", line 202, in call
[00000422] [02-03-2023 08:52:02]     return getattr(self, "chunk_" + cid.decode("ascii"))(pos, length)
[00000423] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\PIL\PngImagePlugin.py", line 568, in chunk_zTXt
[00000424] [02-03-2023 08:52:02]     v = _safe_zlib_decompress(v[1:])
[00000425] [02-03-2023 08:52:02]   File "X:\AI\SDGUI-1.9.0\Data\venv\lib\site-packages\PIL\PngImagePlugin.py", line 148, in _safe_zlib_decompress
[00000426] [02-03-2023 08:52:02]     raise ValueError("Decompressed Data Too Large")
[00000427] [02-03-2023 08:52:02] ValueError: Decompressed Data Too Large