ReppiksProductions
commented
4 days ago I submitted this information over at Kohya's script Github and got a reply from him. It seems that the GUI caches the text encoder and that prevents the text file from being updated during training. Here is his quote:
https://github.com/kohya-ss/sd-scripts/issues/1759#issuecomment-2481248430
When text encoder caching is enabled, the prompts for the sample images are also cached to save memory. Therefore, you cannot change the caption during training. I would like to update it so that training continues while displaying the error.
I haven't found an option to enable or disable "text encoder caching" in the GUI. I would suggest @bmaltais update the GUI under the Sample Prompt section to either exclude "To modify sample prompts during training, edit the prompt.txt file" in the samples directory. or give the option to enable or disable the text encoder caching under this section if it is not available elsewhere.
Greetings,
My LORA training has been interrupted twice now, given the same error. I think I may know the reason it is caused, but not sure. Both times it was interrupted, was after I updated the promp.txt file and it started generating the next sample images. I was looking through the generation and 2 of my prompts didn't have the correct formatting, (Not having a space between a period and the option --w) causing those two prompt to generate with undesirable widths. (512) It happened twice because I never updated the prompt in the GUI, so when I loaded the weights of my last saved LORA and started training again, the GUI overwrote my changes to the prompt.txt. I edited the prompt.txt file again this morning and the same error occurred.
Example my prompt line ended like: ...with palm trees and the ocean in the background.--w 768 --h 1024 --d 777 --s 30
and I changed it to: ...with palm trees and the ocean in the background. --w 768 --h 1024 --d 777 --s 30
The crashes happened when the exact prompt that was changed was set to generate, witch was the 4th prompt out of 5 total. The first time it crashed was during the 2nd set of sample generations, and the second crash was during the 6th set.
Here is my first traceback, I can post the second if needed:
Using cached text encoder outputs for prompt: The image is a close-up portrait of Ptyn1_0 sitting in the back seat of acar. She is smiling and looking directly at the camera. She is wearing a white tank. The car interior is visible through the window, and there are other cars visible in the background. Encoding prompt: The image is a close-up portrait of Ptyn1_0 sitting in the back seat of a car. She is smiling and looking directly at the camera. She is wearing a white tank. The car interior is visible through the window, and there are other cars visible in the background. 100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [03:13<00:00, 6.44s/it] 2024-11-03 01:13:46 INFO prompt: Full body portrait photograph of Ptyn1_0 wearing a pastel flux_train_utils.py:175 bikini, she has blonde hair and blue eyes, she is barefoot, standing on an empty beach, with palm trees and the ocean in the background. INFO height: 1024 flux_train_utils.py:177 INFO width: 768 flux_train_utils.py:178 INFO sample_steps: 30 flux_train_utils.py:179 INFO scale: 3.5 flux_train_utils.py:180 INFO seed: 777 flux_train_utils.py:183 Encoding prompt: Full body portrait photograph of Ptyn1_0 wearing a pastel bikini, she has blonde hair and blue eyes, she is barefoot, standing on an empty beach, with palm trees and the ocean in the background. Traceback (most recent call last): File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\flux_train_network.py", line 564, in <module> trainer.train(args) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\train_network.py", line 1302, in train self.sample_images(accelerator, args, epoch + 1, global_step, accelerator.device, vae, tokenizers, text_encoder, unet) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\flux_train_network.py", line 315, in sample_images flux_train_utils.sample_images( File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\library\flux_train_utils.py", line 89, in sample_images sample_image_inference( File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\library\flux_train_utils.py", line 224, insample_image_inference t5_attn_mask = t5_attn_mask.to(accelerator.device) if args.apply_t5_attn_mask else None AttributeError: 'NoneType' object has no attribute 'to' steps: 13%|██████▊ | 910/6825 [3:23:06<22:00:14, 13.39s/it, avr_loss=0.393] Traceback (most recent call last): File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in <module> sys.exit(main()) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "E:\_python_projects\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['E:\\_python_projects\\Kohya_GUI_Flux_Installer_v42\\kohya_ss\\venv\\Scripts\\python.exe', 'E:/_python_projects/Kohya_GUI_Flux_Installer_v42/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'E:\\_Dropbox\\Dropbox\\AI Source material\\SD\\Created Characters\\Paityn_Hill\\Train_Setup\\model/config_lora-20241102-224749.toml']' returned non-zero exit status 1. 01:13:52-387054 INFO Training has ended.If this is a problem with the actual script files please let me know.