Upstream changes make LoRA extraction from DreamBooth checkpoint impossible

[shadowlocked]

At the advice of the maintainer there, I would be glad if someone here would consider this issue from Kohya_ss. According to the maintainer there, it can only be fixed here.

[kohya-ss]

Could you please share the command line args in console?

[shadowlocked]

Could you please share the command line args in console?

Here is the whole run. As per the linked thread, this results in a 100% non-functional LoRA that has 'lost' the trigger words in the DreamBooth, and produces random imagery, as if there were no prompt supplied.

The installation was updated fully directly before this attempt, including a pip requirements.txt update.

One theory for the issue is that the July 5th and/or May 5th 2023 Diffusers update/s may have moved some of the essential keys and CLIP references, which could be why extracted LoRAs are no longer operative, and can no longer find the trigger words.

Saving the extracted LoRA as Float or FP16 makes no difference. Saving as V2 makes no difference. I have not tried BF16 as I know nothing about it.

_(It is worth mentioning that prior to updating kohya_ss about a week ago, the previous version of kohya was able to successfully create functional LoRAs from older checkpoints created by the Shivam Colab, but not from new checkpoints [the conversion process entirely failed]. Since that update [which solved the problem of catastrophic extraction failure], all kohya-extracted LoRAs of checkpoints from Shivam -- and perhaps all checkpoints from ANY source -- now have zero functional trigger words.)_

---

08:39:42-589150 INFO ./venv/Scripts/python.exe "networks\extract_lora_from_models.py" --save_precision fp16 --save_to "Q:/models/lora/LoraVersion.safetensors" --model_org "C:/Users/USER/Desktop/v1-5-pruned-emaonly.ckpt" --model_tuned "Q:/models/Stable-diffusion/DreamboothModel.ckpt" --dim 128 --device cuda --conv_dim 128 --clamp_quantile 1.0 --min_diff 0.01 loading original SD model : C:/Users/USER/Desktop/v1-5-pruned-emaonly.ckpt UNet2DConditionModel: 64, 8, 768, False, False loading u-net: loading vae: loading text encoder: loading tuned SD model : Q:/models/Stable-diffusion/DreamboothModel.ckpt UNet2DConditionModel: 64, 8, 768, False, False loading u-net: loading vae: loading text encoder: create LoRA network. base dim (rank): 128, alpha: 128 neuron dropout: p=None, rank dropout: p=None, module dropout: p=None apply LoRA to Conv2d with kernel size (3,3). dim (rank): 128, alpha: 128.0 create LoRA for Text Encoder: create LoRA for Text Encoder: 72 modules. create LoRA for U-Net: 278 modules. create LoRA network. base dim (rank): 128, alpha: 128 neuron dropout: p=None, rank dropout: p=None, module dropout: p=None apply LoRA to Conv2d with kernel size (3,3). dim (rank): 128, alpha: 128.0 create LoRA for Text Encoder: create LoRA for Text Encoder: 72 modules. create LoRA for U-Net: 278 modules. Text encoder is same. Extract U-Net only. calculating by svd 100%|██████████████████████| 278/278 [00:35<00:00, 7.85it/s] create LoRA network from weights create LoRA for Text Encoder: create LoRA for Text Encoder: 0 modules. create LoRA for U-Net: 278 modules. enable LoRA for text encoder enable LoRA for U-Net Loading extracted LoRA weights: LoRA weights are saved to: Q:/models/lora/LoraVersion.safetensors

[neojam]

Are "--clamp_quantile" and "--min_diff 0.01" supported in "extract_lora_from_models.py"? When I run help, i dont see those params...

usage: extract_lora_from_models.py [-h] [--v2] [--save_precision {None,float,fp16,bf16}] [--model_org MODEL_ORG] [--model_tuned MODEL_TUNED]
                                   [--save_to SAVE_TO] [--dim DIM] [--conv_dim CONV_DIM] [--device DEVICE]

options:
  -h, --help            show this help message and exit
  --v2                  load Stable Diffusion v2.x model / Stable Diffusion 2.xのモデルを読み込む
  --save_precision {None,float,fp16,bf16}
                        precision in saving, same to merging if omitted / 保存時に精度を変更して保存する、省略時はfloat
  --model_org MODEL_ORG
                        Stable Diffusion original model: ckpt or safetensors file / 元モデル、ckptまたはsafetensors
  --model_tuned MODEL_TUNED
                        Stable Diffusion tuned model, LoRA is difference of `original to tuned`: ckpt or safetensors file /
                        派生モデル（生成されるLoRAは元→派生の差分になります）、ckptまたはsafetensors
  --save_to SAVE_TO     destination file name: ckpt or safetensors file / 保存先のファイル名、ckptまたはsafetensors
  --dim DIM             dimension (rank) of LoRA (default 4) / LoRAの次元数（rank）（デフォルト4）
  --conv_dim CONV_DIM   dimension (rank) of LoRA for Conv2d-3x3 (default None, disabled) / LoRAのConv2d-3x3の次元数（rank）（デフォルトNone、適用なし ）
  --device DEVICE       device to use, cuda for GPU / 計算を行うデバイス、cuda でGPUを使う

[kohya-ss]

--clamp_quantile and --min_diff are not supported in networks\extract_lora_from_models.py. If you are using modified version, please try to use the original version.

[shadowlocked]

Does this script from July not solve the issue? It just needs integrating.

[Sniper199999]

any updates on this issue?

[Mozoloa]

I'm having a similar problem, it looks like the lora extraction fails altho there's no error, the produced barely change the image (if at all). The DB models i'm trying to extract were made with Joe Penna's DB repo

[FugueSegue]

This is extremely distressing for me. For months I've been experimenting and perfecting Dreambooth trainings with the intention of extracting LoRAs. And now I find out that LoRA extraction is completely broken. Not only that but it seems that it has been broken for months and no one has fixed this.

If this isn't going to be fixed then the LoRA extraction tool should be REMOVED. It would be nice if it was fixed but since that isn't going to happen, please remove it so people like myself don't waste hundreds of hours of labor.

[RayHell]

Yes I agree, there's a ton of loss with the extraction. I only tried SDXL dreambooth models.

[JohnsonCobra]

Set min_diff to 0.0001 and voila 🎉

https://github.com/kohya-ss/sd-scripts/issues/1034#issuecomment-1878600892