An error during merging

miasik commented 6 months ago

Tried different models with the same result.

debugs =  ['elemental merge']
use_extra_elements =  True
 - mm_max_models =  4
config hash =  f79746d04db933c7fd3d9ae6cb70b3bd562a09ed41e48bc894b4fec58d5259e1
  - mm_use [True, False, False, False]
  - model_a Photo\nextphoto_v30.safetensors [1c1f913f3b]
  - base_model v1-5-pruned-emaonly.safetensors [6ce0161689]
  - max_models 4
  - models ['Photo\\rundiffusionFX_v10.safetensors [ad1a10552b]']
  - modes ['Add-Diff']
  - calcmodes ['Rebasin']
  - usembws [[]]
  - weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
  - alpha [0.75]
  - adjust 1,1,1,0,0,0,0,0
  - use elemental [False]
  - elementals ['']
  - Parse elemental merge...
model_a = Photo_nextphoto_v30
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Photo\nextphoto_v30.safetensors...
isxl = False
compact_mode =  False
 - Dynamic loading rebasin module...
Rebasin mode
 - Calulation device for Rebasin is  cuda
 - LAP library is lap
Loading model Photo_rundiffusionFX_v10...
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Photo\rundiffusionFX_v10.safetensors...
mode = Add-Diff, alpha = 0.75
Stage #1/2: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1132/1132 [00:15<00:00, 74.62it/s]
Check uninitialized #2/2: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1132/1132 [00:00<00:00, 189153.90it/s]
Rebasin calc...
maximize weight matching using scipy linear_sum_assignment...
P_bg337:  24%|██████████Special layer P_bg337 found███████████████▉                                                                                                                                                                       | 125/519 [00:12<00:25, 15.58it/s]
P_bg265: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 519/519 [00:38<00:00, 13.33it/s]
P_bg176: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 519/519 [00:35<00:00, 14.43it/s]
P_bg176: new - old = 0.0
weight order changed layers = ['P_bg371', 'P_bg324', 'P_bg337', 'P_bg358']
Save unchanged weights #2/2: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]
Apply fine tune [0.99, 1.02, 0.99, 1.02, 0.99, [0.0, 0.0, 0.0, 0.0]]
Clip is fine
 - merge processing in 95.3s (prepare: 0.4s, merging: 94.9s).
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3746, in before_process
        sd_models.send_model_to_cpu(sd_models.model_data.sd_model)
      File "E:\SD\automatic1111\modules\sd_models.py", line 576, in send_model_to_cpu
        m.to(devices.cpu)
      File "e:\SD\automatic1111\venv\lib\site-packages\lightning_fabric\utilities\device_dtype_mixin.py", line 54, in to
        return super().to(*args, **kwargs)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 1145, in to
        return self._apply(convert)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      [Previous line repeated 1 more time]
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 820, in _apply
        param_applied = fn(param)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 1143, in convert
        return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
    NotImplementedError: Cannot copy out of meta tensor; no data!

---
*** Error completing request
*** Arguments: ('task(26bs7tv83v5lryc)', 'A real life analog photo of a provocative adult woman,\nred lips, camisole, low leg jeans shorts, boobs drop, side boobs, groin,\njewelry,\ndetailed skin, skin pores, (freckles:0.6), (moles:0.6), (pigmentation:0.5),\ncatwalking on a street at night,\n[[Gabrielle Union | Laura Vandervoort]:0.1],\nphotography achievement, depth of field, film grain', '(worst quality:1.3), (bad quality:1.2), (low quality:1.1),\n[deformed | disfigured], poorly drawn, [bad : wrong] anatomy, [extra | missing | floating | disconnected] limb, (mutated hands and fingers)', [], 32, 'DPM++ 3M SDE Karras', 1, 1, 7, 768, 512, False, 0.3, 2, '4x_NMKD-Siax_200k', 24, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x0000029F347CFBB0>, 0, False, '', 0.8, 1518790282, False, -1, 0, 0, 0, False, False, {'ad_model': 'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M Karras', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 32, 'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'DPM++ 2M Karras', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, False, 7, 100, 'Constant', 0, 'Constant', 0, 4, True, 'MEAN', 'AD', 1, False, 1, 1, 0, 0, -1, 0, 0, 0, 0, 0, True, -1, 1, 0, '1,1', 'Horizontal', '', 2, 1, True, 'Photo\\nextphoto_v30.safetensors [1c1f913f3b]', 'v1-5-pruned-emaonly.safetensors [6ce0161689]', 4, '1,1,1,0,0,0,0,0', {'calcmodes': ('Rebasin', 'Normal', 'Normal', 'Normal'), 'save_settings': ['fp16', 'prune', 'safetensors'], 'calc_settings': ['GPU']}, True, False, False, False, 'Photo\\rundiffusionFX_v10.safetensors [ad1a10552b]', 'None', 'None', 'None', 'Add-Diff', 'Sum', 'Sum', 'Sum', 0.75, 0.5, 0.5, 0.5, False, True, True, True, [], [], [], [], [], [], [], [], '0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', '0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', '0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', '0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5', False, False, False, False, '', '', '', '', False, False, 'positive', 'comma', 0, False, False, 'start', '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False) {}
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "E:\SD\automatic1111\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "E:\SD\automatic1111\modules\txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "E:\SD\automatic1111\modules\processing.py", line 721, in process_images
        sd_models.reload_model_weights()
      File "E:\SD\automatic1111\modules\sd_models.py", line 764, in reload_model_weights
        sd_model = reuse_model_from_already_loaded(sd_model, checkpoint_info, timer)
      File "E:\SD\automatic1111\modules\sd_models.py", line 713, in reuse_model_from_already_loaded
        send_model_to_cpu(sd_model)
      File "E:\SD\automatic1111\modules\sd_models.py", line 576, in send_model_to_cpu
        m.to(devices.cpu)
      File "e:\SD\automatic1111\venv\lib\site-packages\lightning_fabric\utilities\device_dtype_mixin.py", line 54, in to
        return super().to(*args, **kwargs)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 1145, in to
        return self._apply(convert)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
        module._apply(fn)
      [Previous line repeated 1 more time]
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 820, in _apply
        param_applied = fn(param)
      File "e:\SD\automatic1111\venv\lib\site-packages\torch\nn\modules\module.py", line 1143, in convert
        return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
    NotImplementedError: Cannot copy out of meta tensor; no data!

---

wkpark commented 6 months ago

Thank you for your reporting, I have the same issue with the recent 1.7.x sd-webui (currently, I'm using sd-webui dev branch)

Please check some settings "Settings"->"Optimization" and turn off "FP8" optimization (Enable to Disable)

miasik commented 6 months ago

Thank you for your reporting, I have the same issue with the recent 1.7.x sd-webui (currently, I'm using sd-webui dev branch)

Please check some settings "Settings"->"Optimization" and turn off "FP8" optimization (Enable to Disable)

I don't see/find such an option

wkpark commented 6 months ago

I guess this is a similar issue. https://github.com/wkpark/sd-webui-model-mixer/pull/29

miasik commented 6 months ago

I guess this is a similar issue. #29 Should I add more logs?

miasik commented 6 months ago

It looks working now for Sum mode, but Add-Diff/Rebasing still does not. It shows its progress and a1111 renders a picture in this mode thу picture never changes. If I change mode the picture follows corresponding merges but it freezes again if I switch back to Add-Diff/Rebasing. Also, what Rebasing and Fast Rb are?

debugs =  ['elemental merge', 'merge', 'adjust']█████████████████                                            | 32/64 [00:08<00:08,  3.85it/s]
use_extra_elements =  True
 - mm_max_models =  4
config hash =  73453409346ffcb55fca326b0e055e898aef8cacd938fcbc304fd6e4a1143e13
  - mm_use [True, False, False, False]
  - model_a Cartoons\fluffity_v21.safetensors [68536065c9]
  - base_model v1-5-pruned-emaonly.safetensors [6ce0161689]
  - max_models 4
  - models ['Photo\\epicphotogasm_lastUnicorn.safetensors [62bb78983a]']
  - modes ['Add-Diff']
  - calcmodes ['Rebasin']
  - usembws [[]]
  - weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
  - alpha [0.25]
  - adjust 1,1,1,0,0,0,0,0
  - use elemental [False]
  - elementals ['']
  - Parse elemental merge...
model_a = Cartoons_fluffity_v21
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Cartoons\fluffity_v21.safetensors...
isxl = False
compact_mode =  False
 - check possible UNet partial update...
 - partial changed blocks =  ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'IN09', 'IN10', 'IN11', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08', 'OUT09', 'OUT10', 'OUT11']
 - UNet partial update mode
 - Dynamic loading rebasin module...
Rebasin mode
 - Calulation device for Rebasin is  cuda
 - LAP library is lap
Loading model Photo_epicphotogasm_lastUnicorn...
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Photo\epicphotogasm_lastUnicorn.safetensors...
mode = Add-Diff, alpha = 0.25
Stage #1/2: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 876/876 [00:03<00:00, 237.75it/s]
Check uninitialized #2/2: 100%|████████████████████████████████████████████████████████████████████████| 876/876 [00:00<00:00, 146382.88it/s]
Rebasin calc...
maximize weight matching using scipy linear_sum_assignment...
P_bg324: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 12.03it/s]
P_bg324: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 11.53it/s]
P_bg324: new - old = 0.0
weight order changed layers = ['P_bg358', 'P_bg371', 'P_bg324']
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3482, in before_process
        theta_0 = apply_permutation(permutation_spec, first_permutation, theta_0)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 833, in apply_permutation
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if "model_" not in k}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 833, in <dictcomp>
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if "model_" not in k}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 773, in get_permuted_param
        for axis, p in enumerate(ps.axes_to_perm[k]):
    KeyError: 'cond_stage_model.logit_scale'

---
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:08<00:00,  3.66it/s]
Total progress:  50%|████████████████████████████████████████████                                            | 32/64 [00:08<00:08,  3.71it/s]
Total progress:  50%|████████████████████████████████████████████                                            | 32/64 [00:08<00:08,  3.89it/s]

wkpark commented 6 months ago

It looks working now for Sum mode, but Add-Diff/Rebasing still does not. It shows its progress and a1111 renders a picture in this mode thу picture never changes. If I change mode the picture follows corresponding merges but it freezes again if I switch back to Add-Diff/Rebasing. Also, what Rebasing and Fast Rb are?

debugs =  ['elemental merge', 'merge', 'adjust']█████████████████                                            | 32/64 [00:08<00:08,  3.85it/s]
use_extra_elements =  True
 - mm_max_models =  4
config hash =  73453409346ffcb55fca326b0e055e898aef8cacd938fcbc304fd6e4a1143e13
  - mm_use [True, False, False, False]
  - model_a Cartoons\fluffity_v21.safetensors [68536065c9]
  - base_model v1-5-pruned-emaonly.safetensors [6ce0161689]
  - max_models 4
  - models ['Photo\\epicphotogasm_lastUnicorn.safetensors [62bb78983a]']
  - modes ['Add-Diff']
  - calcmodes ['Rebasin']
  - usembws [[]]
  - weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
  - alpha [0.25]
  - adjust 1,1,1,0,0,0,0,0
  - use elemental [False]
  - elementals ['']
  - Parse elemental merge...
model_a = Cartoons_fluffity_v21
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Cartoons\fluffity_v21.safetensors...
isxl = False
compact_mode =  False
 - check possible UNet partial update...
 - partial changed blocks =  ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'IN09', 'IN10', 'IN11', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08', 'OUT09', 'OUT10', 'OUT11']
 - UNet partial update mode
 - Dynamic loading rebasin module...
Rebasin mode
 - Calulation device for Rebasin is  cuda
 - LAP library is lap
Loading model Photo_epicphotogasm_lastUnicorn...
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Photo\epicphotogasm_lastUnicorn.safetensors...
mode = Add-Diff, alpha = 0.25
Stage #1/2: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 876/876 [00:03<00:00, 237.75it/s]
Check uninitialized #2/2: 100%|████████████████████████████████████████████████████████████████████████| 876/876 [00:00<00:00, 146382.88it/s]
Rebasin calc...
maximize weight matching using scipy linear_sum_assignment...
P_bg324: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 12.03it/s]
P_bg324: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 11.53it/s]
P_bg324: new - old = 0.0
weight order changed layers = ['P_bg358', 'P_bg371', 'P_bg324']
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3482, in before_process
        theta_0 = apply_permutation(permutation_spec, first_permutation, theta_0)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 833, in apply_permutation
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if "model_" not in k}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 833, in <dictcomp>
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if "model_" not in k}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 773, in get_permuted_param
        for axis, p in enumerate(ps.axes_to_perm[k]):
    KeyError: 'cond_stage_model.logit_scale'

---
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:08<00:00,  3.66it/s]
Total progress:  50%|████████████████████████████████████████████                                            | 32/64 [00:08<00:08,  3.71it/s]
Total progress:  50%|████████████████████████████████████████████                                            | 32/64 [00:08<00:08,  3.89it/s]

Rebasin only support SD1.5 and does not support SD2.1 / SDXL currently. cond_stage_model.logit_scale key is not for SD1.5

"Fast rebasin" only checks special weights groups. (rebasin itself checks all permutation groups and sorts keys internally results possible permutations. but almost all orders of permutation groups are not changed (identical) except special groups. so Fast Rebasin will result identical results if it does not found special groups on merging blocks)

miasik commented 6 months ago

Rebasin only support SD1.5 and does not support SD2.1 / SDXL currently. cond_stage_model.logit_scale key is not for SD1.5

It couldn't be the reason, because both models are 1.5 — they were posted as 1.5 and they look fully functional as 1.5 models. Actually, I can merge them as 1.5 models by Sum mode. Is there a way to just skip unexpected keys and continue merging?

wkpark commented 6 months ago

Rebasin only support SD1.5 and does not support SD2.1 / SDXL currently. cond_stage_model.logit_scale key is not for SD1.5

It couldn't be the reason, because both models are 1.5 — they were posted as 1.5 and they look fully functional as 1.5 models. Actually, I can merge them as 1.5 models by Sum mode. Is there a way to just skip unexpected keys and continue merging?

OK. I see, It needs to be checked the validity or acceptable keys in the weight matching routine.

wkpark commented 6 months ago

This issue has been automatically closed. Please reopen if this issue is not resolved

miasik commented 6 months ago

As I see — nothing has changed. Still getting the same error and the rendered picture doesn't chande

*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3487, in before_process
        theta_0 = apply_permutation(permutation_spec, first_permutation, theta_0)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 839, in apply_permutation
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if _valid_key(k)}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 839, in <dictcomp>
        return {k: get_permuted_param(ps, perm, k, params) for k in params.keys() if _valid_key(k)}
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\rebasin\weight_matching.py", line 773, in get_permuted_param
        for axis, p in enumerate(ps.axes_to_perm[k]):
    KeyError: 'cond_stage_model.logit_scale'

wkpark commented 6 months ago

thanks for your reporting. I've fixed logic bug again

miasik commented 6 months ago

Nothing new again. Rebasin doesn't change the image

debugs =  ['elemental merge', 'merge', 'adjust']████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:12<00:00,  2.44it/s]
use_extra_elements =  True
 - mm_max_models =  4
config hash =  0abd1b6d1456626614caacd09c282b776869dcac07788f0289caa9e1a80dbfb5
  - mm_use [True, False, False, False]
  - model_a Cartoons\fluffity_v21.safetensors [68536065c9]
  - base_model Cartoons\fluffity_v21.safetensors [68536065c9]
  - max_models 4
  - models ['Photo\\epicphotogasm_lastUnicorn.safetensors [62bb78983a]']
  - modes ['Add-Diff']
  - calcmodes ['Rebasin']
  - usembws [[]]
  - weights ['0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5']
  - alpha [0.01]
  - adjust
  - use elemental [False]
  - elementals ['']
  - Parse elemental merge...
model_a = Cartoons_fluffity_v21
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Cartoons\fluffity_v21.safetensors...
isxl = False , sd2 = False
compact_mode =  False
 - check possible UNet partial update...
 - partial changed blocks =  ['BASE', 'IN00', 'IN01', 'IN02', 'IN03', 'IN04', 'IN05', 'IN06', 'IN07', 'IN08', 'IN09', 'IN10', 'IN11', 'M00', 'OUT00', 'OUT01', 'OUT02', 'OUT03', 'OUT04', 'OUT05', 'OUT06', 'OUT07', 'OUT08', 'OUT09', 'OUT10', 'OUT11']
 - UNet partial update mode
 - Dynamic loading rebasin module...
Rebasin mode
 - Calulation device for Rebasin is  cuda
 - LAP library is lap
Loading model Photo_epicphotogasm_lastUnicorn...
Loading from file E:\SD\automatic1111\models\Stable-diffusion\Photo\epicphotogasm_lastUnicorn.safetensors...
mode = Add-Diff, alpha = 0.01
Stage #1/2: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 876/876 [00:13<00:00, 62.83it/s]
Check uninitialized #2/2: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 876/876 [00:00<00:00, 125476.75it/s]
Rebasin calc...
maximize weight matching using scipy linear_sum_assignment...
P_bg324: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 10.52it/s]
P_bg324: new - old = 0.0
weight order changed layers = ['P_bg358', 'P_bg371', 'P_bg324']
Save unchanged weights #2/2: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]
Clip is fine
 - merge processing in 17.5s (prepare: 0.5s, merging: 17.1s).
 - loading scripts.patches...
 - Fail to patch lora
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 3684, in before_process
        base_dict[key] = theta_0[k]
    KeyError: 'cond_stage_model.logit_scale'

---
Using already loaded model Cartoons_fluffity_v21 + (Photo_epicphotogasm_lastUnicorn - v1-5-pruned-emaonly.safetensors [6ce0161689]) x alpha_0(0.25).safetensors [955683620d]: done in 0.5s (send model to device: 0.5s)
Using already loaded model Cartoons_fluffity_v21 + (Photo_epicphotogasm_lastUnicorn - v1-5-pruned-emaonly.safetensors [6ce0161689]) x alpha_0(0.25).safetensors [955683620d]: done in 0.0s
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:12<00:00,  2.37it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:12<00:00,  2.37it/s]
Total progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:12<00:00,  2.43it/s]

wkpark commented 6 months ago

OK. this is another bug. rebasin works just fine. fixed another bug found at the partial update routine.

miasik commented 6 months ago

Ok, this time i noticed some minor changes and I cant get if it's OK I set XYZ plot with [Model Mixer] alpha B for X and 0,0.25,0.5,0.75,1 as parameter Then I got a grid but the changes almost invisible

New errors, one for each X value:

debugs =  ['elemental merge', 'merge', 'adjust']█▍                                                                                                                 | 30/150 [00:07<00:28,  4.26it/s]
use_extra_elements =  True
 - mm_max_models =  4
XYZ: alpha b -> 0.25
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 2756, in before_process
        mm_finetune = ",".join([str(int(float(x))) if float(x) == int(float(x)) else str(x) for x in fines])
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 2756, in <listcomp>
        mm_finetune = ",".join([str(int(float(x))) if float(x) == int(float(x)) else str(x) for x in fines])
    ValueError: could not convert string to float: ''

wkpark commented 6 months ago

Ok, this time i noticed some minor changes and I cant get if it's OK I set XYZ plot with [Model Mixer] alpha B for X and 0,0.25,0.5,0.75,1 as parameter Then I got a grid but the changes almost invisible

New errors, one for each X value:

debugs =  ['elemental merge', 'merge', 'adjust']█▍                                                                                                                 | 30/150 [00:07<00:28,  4.26it/s]
use_extra_elements =  True
 - mm_max_models =  4
XYZ: alpha b -> 0.25
*** Error running before_process: E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py
    Traceback (most recent call last):
      File "E:\SD\automatic1111\modules\scripts.py", line 710, in before_process
        script.before_process(p, *script_args)
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 2756, in before_process
        mm_finetune = ",".join([str(int(float(x))) if float(x) == int(float(x)) else str(x) for x in fines])
      File "E:\SD\automatic1111\extensions\sd-webui-model-mixer\scripts\model_mixer.py", line 2756, in <listcomp>
        mm_finetune = ",".join([str(int(float(x))) if float(x) == int(float(x)) else str(x) for x in fines])
    ValueError: could not convert string to float: ''

thanks again. you just found another bug again.

and fixed now

miasik commented 6 months ago

Fantastic! ;-) I see my new grid and it looks like we've finally done here :-D Thank you so much!

Next I want to ensure that Rebasin works not worse than Train Difference. Because TD is amazing.

wkpark commented 6 months ago

Thank you for your patience and quick feedback!

wkpark / sd-webui-model-mixer

An error during merging #95