search

hako-mikan / sd-webui-supermerger

model merge extention for stable diffusion web ui
GNU Affero General Public License v3.0
753 stars 111 forks source link

2つのモデル差分からの差分LoRA作成に失敗する #318

Closed JujoHotaru closed 9 months ago

JujoHotaru commented 11 months ago

いつもお世話になっております。

2つのcheckpointからの差分LoRA作成機能が、WebUI1.7＋最新版の環境で動作しなくなっていました。

最初に気付いたのは別の作業中でしたが、再現手順として、当方で公開している2つのモデルバリエーション（AngelBreed_Tsurime_v20とAngelBreed_v20）の差分を出してみようとすると発生します。

make LoRA start
Loading weights [2d9b151571] from C:\stable-diffusion-webui\models\Stable-diffusion\AngelBreed_Tsurime_v20.safetensors
Creating model from config: C:\stable-diffusion-webui\configs\v1-inference.yaml
Loading VAE weights specified in settings: C:\stable-diffusion-webui\models\VAE\clearvae_main.safetensors
Applying attention optimization: sdp-no-mem... done.
Model loaded in 1.7s (unload existing model: 0.1s, load weights from disk: 0.2s, create model: 0.2s, apply weights to model: 0.8s, load VAE: 0.2s).
Detected model type: SDXL: False, SD2.X: False, SD1.X: True
loading original SD model : C:\stable-diffusion-webui\models\Stable-diffusion\AngelBreed_Tsurime_v20.safetensors
UNet2DConditionModel: 64, 8, 768, False, False
loading u-net: <All keys matched successfully>
Traceback (most recent call last):
  File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "C:\stable-diffusion-webui\extensions\sd-webui-supermerger\scripts\mergers\pluslora.py", line 293, in makelora
    result = ext.svd(args)
  File "C:\stable-diffusion-webui\extensions\sd-webui-supermerger\scripts\kohyas\extract_lora_from_models.py", line 52, in svd
    text_encoder_o, _, unet_o = model_util.load_models_from_stable_diffusion_checkpoint(args.v2, args.model_org)
  File "C:\stable-diffusion-webui\extensions\sd-webui-supermerger\scripts\kohyas\model_util.py", line 1015, in load_models_from_stable_diffusion_checkpoint
    info = vae.load_state_dict(converted_vae_checkpoint)
  File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for AutoencoderKL:
        Unexpected key(s) in state_dict: "encoder.mid_block.attentions.0.to_key.bias", "encoder.mid_block.attentions.0.to_key.weight", "encoder.mid_block.attentions.0.to_out.0.bias", "encoder.mid_block.attentions.0.to_out.0.weight", "encoder.mid_block.attentions.0.to_query.bias", "encoder.mid_block.attentions.0.to_query.weight", "encoder.mid_block.attentions.0.to_value.bias", "encoder.mid_block.attentions.0.to_value.weight", "decoder.mid_block.attentions.0.to_key.bias", "decoder.mid_block.attentions.0.to_key.weight", "decoder.mid_block.attentions.0.to_out.0.bias", "decoder.mid_block.attentions.0.to_out.0.weight", "decoder.mid_block.attentions.0.to_query.bias", "decoder.mid_block.attentions.0.to_query.weight", "decoder.mid_block.attentions.0.to_value.bias", "decoder.mid_block.attentions.0.to_value.weight".

手元で保存してある古い環境では発生しませんでした。

JujoHotaru commented 11 months ago

ログを見ていて気付いたのですが、旧環境ではcheckpoint Aとcheckpoint Bの両方のファイル名がログに出てくるのですが、上記エラー状況ですと、checkpoint Bのモデル（AngelBreed_v20）がロードされた形跡がない？ように見えます。

旧環境で成功した場合のログ：

make LoRA start
loading SD model : E:\stable-diffusion-webui_152\models\Stable-diffusion\AngelBreed_v20.safetensors
diffusers version :  0.20.1
loading u-net: <All keys matched successfully>
loading vae: failed
loading text encoder: <All keys matched successfully>
loading SD model : E:\stable-diffusion-webui_152\models\Stable-diffusion\AngelBreed_Tsurime_v20.safetensors
diffusers version :  0.20.1
loading u-net: <All keys matched successfully>
loading vae: failed
loading text encoder: <All keys matched successfully>
create LoRA network. base dim (rank): 16, alpha: 16
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 128 modules.
create LoRA network. base dim (rank): 16, alpha: 16
create LoRA for Text Encoder: 72 modules.
create LoRA for U-Net: 128 modules.
Text encoder is same. Extract U-Net only.
calculating by svd
100%|████████████████████████████████████████████████████████████████████████████████| 128/128 [00:13<00:00,  9.32it/s]
enable LoRA for U-Net
LoRA has 384 weights.
Loading extracted LoRA weights: <All keys matched successfully>
LoRA weights are saved to: E:\stable-diffusion-webui_152\models\Lora\difftest.safetensors
silveroxides commented 11 months ago

If you look at the newer log you can see that it loads a VAE. Also the keys it is mentioning at last line are VAE elements without their prefixes which is how they are when VAE is not baked into model. So for some reason it is detecting the VAE loaded and they mismatch since it expects the prefix 'first_stage_model.'

hako-mikan commented 10 months ago

VAEの読み込み時のエラーのようなので、VAEを読み込まないよう修正しました。

JujoHotaru commented 9 months ago

ご対応ありがとうございました！LoRA作成できているようです。