Extract_lora_from_models.py hasn't worked in a while

[CCpt5]

Hello! Thanks so much for your work on this project - it's incredible and a complete necessity.

I often use dreambooth to train and then extra LoRAs from the trained models afterward. To do that I use your extra_lora_from_models.py script. I noticed months back that it had stopped working properly for me where generated LoRA wouldn't make any changed when trigger words were used. So at that point I rolled back and ended up keeping the script dated 2023-08-22. I recently recreated my full training folder and w/ that pulled the latest version of the script. As far as I can tell this issue still persists.

Here is a console log below which is an attempted extra of a LoRA from an SVCD checkpoint 1st w/ the latest version and after w/ the 2023-08-22 version. I noticed the new version says, "Text encoder is same. Extract U-Net only." while the older version that works for me says, " Text encoder is different. 0.0032296180725097656 > 0.0001"

I hope you can give the current code a look to make sure nothing is improper.

Thanks!!

D:\kohya_ss>.\venv\scripts\activate

(venv) D:\kohya_ss>cd networks

(venv) D:\kohya_ss\networks>python extract_lora_from_models.py --sdxl --save_precision="bf16" --model_org="d:\Kohya-Dev\sd-scripts\training_models\realvis.safetensors" --model_tuned="e:\Stable Diffusion Checkpoints\SDXL\2024-01-02 - Emma Corrin SDXL - RealVision - 17img (20repeats) - 200reg - b2-step00002100.safetensors" --save_to="d:\stable-diffusion-webui\models\Lora\2024-01-02 - Emma Corrin SDXL - RealVision - 17img (20repeats) - 200reg - b2-step00002100.safetensors" --dim=64 --device cuda
loading original SDXL model : d:\Kohya-Dev\sd-scripts\training_models\realvis.safetensors
Traceback (most recent call last):
  File "D:\kohya_ss\networks\extract_lora_from_models.py", line 296, in <module>
    svd(**vars(args))
  File "D:\kohya_ss\networks\extract_lora_from_models.py", line 73, in svd
    text_encoder_o1, text_encoder_o2, _, unet_o, _, _ = sdxl_model_util.load_models_from_sdxl_checkpoint(
  File "D:\kohya_ss\library\sdxl_model_util.py", line 177, in load_models_from_sdxl_checkpoint
    state_dict = load_file(ckpt_path)  # prevent device invalid Error
  File "D:\kohya_ss\venv\lib\site-packages\safetensors\torch.py", line 259, in load_file
    with safe_open(filename, framework="pt", device=device) as f:
FileNotFoundError: No such file or directory: "d:\\Kohya-Dev\\sd-scripts\\training_models\\realvis.safetensors"

(venv) D:\kohya_ss\networks>python extract_lora_from_models.py --sdxl --save_precision="bf16" --model_org="d:\kohya_ss\training_models\realvis.safetensors" --model_tuned="e:\Stable Diffusion Checkpoints\SDXL\2024-01-02 - Emma Corrin SDXL - RealVision - 17img (20repeats) - 200reg - b2-step00002100.safetensors" --save_to="d:\stable-diffusion-webui\models\Lora\2024-01-02 - Emma Corrin SDXL - RealVision - 17img (20repeats) - 200reg - b2-step00002100.safetensors" --dim=64 --device cuda
loading original SDXL model : d:\kohya_ss\training_models\realvis.safetensors
building U-Net
loading U-Net from checkpoint
U-Net:  <All keys matched successfully>
building text encoders
loading text encoders from checkpoint
text encoder 1: <All keys matched successfully>
text encoder 2: <All keys matched successfully>
building VAE
loading VAE from checkpoint
VAE: <All keys matched successfully>
loading original SDXL model : e:\Stable Diffusion Checkpoints\SDXL\2024-01-02 - Emma Corrin SDXL - RealVision - 17img (20repeats) - 200reg - b2-step00002100.safetensors
building U-Net
loading U-Net from checkpoint
U-Net:  <All keys matched successfully>
building text encoders
loading text encoders from checkpoint
text encoder 1: <All keys matched successfully>
text encoder 2: <All keys matched successfully>
building VAE
loading VAE from checkpoint
VAE: <All keys matched successfully>
create LoRA network. base dim (rank): 64, alpha: 64
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder 1:
create LoRA for Text Encoder 2:
create LoRA for Text Encoder: 264 modules.
create LoRA for U-Net: 722 modules.
create LoRA network. base dim (rank): 64, alpha: 64
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder 1:
create LoRA for Text Encoder 2:
create LoRA for Text Encoder: 264 modules.
create LoRA for U-Net: 722 modules.
Text encoder is same. Extract U-Net only.
calculating by svd
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 722/722 [00:49<00:00, 14.73it/s]
create LoRA network from weights
create LoRA for Text Encoder 1:
create LoRA for Text Encoder 2:
create LoRA for Text Encoder: 0 modules.
create LoRA for U-Net: 722 modules.
enable LoRA for text encoder
enable LoRA for U-Net
Loading extracted LoRA weights: <All keys matched successfully>
LoRA weights are saved to: d:\stable-diffusion-webui\models\Lora\2024-01-02 - Emma Corrin SDXL - RealVision - 17img (20repeats) - 200reg - b2-step00002100.safetensors

(venv) D:\kohya_ss\networks>python extract_lora_from_models.py --sdxl --save_precision="bf16" --model_org="d:\kohya_ss\training_models\realvis.safetensors" --model_tuned="e:\Stable Diffusion Checkpoints\SDXL\2024-01-02 - Emma Corrin SDXL - RealVision - 17img (20repeats) - 200reg - b2-step00002100.safetensors" --save_to="d:\stable-diffusion-webui\models\Lora\2024-01-02 - Emma Corrin SDXL - RealVision - 17img (20repeats) - 200reg - b2-step00002100.safetensors" --dim=64 --device cuda
loading original SDXL model : d:\kohya_ss\training_models\realvis.safetensors
building U-Net
loading U-Net from checkpoint
U-Net:  <All keys matched successfully>
building text encoders
loading text encoders from checkpoint
text encoder 1: <All keys matched successfully>
text encoder 2: <All keys matched successfully>
building VAE
loading VAE from checkpoint
VAE: <All keys matched successfully>
loading original SDXL model : e:\Stable Diffusion Checkpoints\SDXL\2024-01-02 - Emma Corrin SDXL - RealVision - 17img (20repeats) - 200reg - b2-step00002100.safetensors
building U-Net
loading U-Net from checkpoint
U-Net:  <All keys matched successfully>
building text encoders
loading text encoders from checkpoint
text encoder 1: <All keys matched successfully>
text encoder 2: <All keys matched successfully>
building VAE
loading VAE from checkpoint
VAE: <All keys matched successfully>
create LoRA network. base dim (rank): 64, alpha: 64
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder 1:
create LoRA for Text Encoder 2:
create LoRA for Text Encoder: 264 modules.
create LoRA for U-Net: 722 modules.
create LoRA network. base dim (rank): 64, alpha: 64
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder 1:
create LoRA for Text Encoder 2:
create LoRA for Text Encoder: 264 modules.
create LoRA for U-Net: 722 modules.
Text encoder is different. 0.0032296180725097656 > 0.0001
calculating by svd
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 986/986 [01:05<00:00, 14.94it/s]
create LoRA network from weights
create LoRA for Text Encoder 1:
create LoRA for Text Encoder 2:
create LoRA for Text Encoder: 264 modules.
create LoRA for U-Net: 722 modules.
enable LoRA for text encoder
enable LoRA for U-Net
Loading extracted LoRA weights: <All keys matched successfully>
LoRA weights are saved to: d:\stable-diffusion-webui\models\Lora\2024-01-02 - Emma Corrin SDXL - RealVision - 17img (20repeats) - 200reg - b2-step00002100.safetensors

[CCpt5]

Here's a comparison of the two .pys using Beyond Compare - maybe helpful.

Current extract lora script vs. Aug '23 version that I used which works:

compare-kohya.pdf

[kohya-ss]

The latest version of the extract script has --min_diff option. If you set the small value like 0.0001 for it, same as the older version, the script will extract Text Encoder part of LoRA.

[CCpt5]

Ok cool, thanks. Wasn't aware that flag was needed. Appreciate it, and maybe this thread will help someone else at least :)