Suggestion for cosine similarity

[Mohamed-Desouki]

Hi , Thanks for this great extension and thanks for the method creators. When I am reading about merges I found on this model ( https://civitai.com/models/75706/chillymixbase), mentioned "total output blocks similarity: 95.4978% and total input blocks similarity: 87.7581% (MID0 seems too low relatively". I like the stable diffusion and I am not good at python; it is a hobby; but I tried to go through your code and with losslessmix code to get a script to calculate the cosine similarity. I made this code:

` import os import argparse import torch import numpy as np from tqdm import tqdm

parser = argparse.ArgumentParser(description="Calculate cosine similarity between models") parser.add_argument("a", type=str, help="Path to model a") parser.add_argument("b", type=str, help="Path to model b") parser.add_argument("--out", type=str, help="Output file name, without extension", default="cosine_similarity", required=False) parser.add_argument("--device", type=str, help="Device to use, defaults to cpu", default="cpu", required=False) args = parser.parse_args()

def loadModelWeights(mPath): model = torch.load(mPath, map_location=args.device) try: theta = model["state_dict"] except: theta = model return theta

a, b = loadModelWeights(args.a), loadModelWeights(args.b)

cosine_similarities = []

for key in tqdm(a.keys(), desc="Calculating cosine similarity"): if key in b and a[key].size() == b[key].size(): a_flat = a[key].view(-1).to(torch.float32) b_flat = b[key].view(-1).to(torch.float32) simab = torch.nn.functional.cosine_similarity(a_flat.unsqueeze(0), b_flat.unsqueeze(0)) cosine_similarities.append((key, simab.item()))

with open(f"lesser({args.a.split('/')[-1]}-{args.b.split('/')[-1]}).txt", 'w') as f1, open(f"greater({args.a.split('/')[-1]}-{args.b.split('/')[-1]}).txt", 'w') as f2: for key, similarity in cosine_similarities: if similarity < 0.9: f1.write(f"{key}: {similarity}\n") else: f2.write(f"{key}: {similarity}\n")

print(f"Done! Cosine similarity written to lesser({args.a.split('/')[-1]}-{args.b.split('/')[-1]}).txt and greater({args.a.split('/')[-1]}-{args.b.split('/')[-1]}).txt.") ` and I got 2 files ( one for weight less than 0.9 and one for equal or greater than 0.9). The point is the the data are too many in the files, and I do not know what is what. How (https://civitai.com/models/75706/chillymixbase) got this, or how he know from the output data. | IN01 | IN02 | IN04 | IN05 | IN07 | IN08 | MID0 | | 93.2390% | 95.7006% | 90.4897% | 92.1093% | 83.8271% | 85.8409% | 73.0998% |

my suggestion is to add an option for getting output text file showing the cosine similarity before the merge, so we can change the models before merge or have an idea about the merged data. Thanks in advance.

[hako-mikan]

Thanks suggestion and code. I added "Analysis" tab to calculate difference. In this tab, I used your code.

in https://civitai.com/models/75706/chillymixbase, they may use ASimilarity, so I also added ASimilarity mode.

[Mohamed-Desouki]

Thanks for your efforts. I found that in another post he mentioned the used extensions. Blessing Mix recipe Supermerger ASimilarityCalculatior Modified ASimilarity. Hope that helps for future development.