N Matching Results - Similarity Scores Too Close for Non-Similar Faces

[utkarsh-chaurasia]

Description: I have tested a set of images using your model weights and performed N matching based on cosine similarity. I followed the code structure below:

Sample code for N:N matching using cosine similarity

import argparse
import os
import cv2
import torch
import torch.nn.functional as F
import pandas as pd
from config.config import config as cfg
from torchvision import transforms
from backbones.iresnet import iresnet50
from sklearn.metrics.pairwise import cosine_similarity

torch.backends.cudnn.benchmark = True

class FaceRecognitionModel:
    def __init__(self, model_path):
        self.model_path = model_path
        self.device = torch.device('cpu')  # Set to CPU
        self.transform = transforms.Compose([
            transforms.ToPILImage(),
            transforms.Resize((112, 112)),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
        ])
        self.backbone = self.load_model()

    def load_model(self):
        # Initialize the model
        backbone = iresnet50(dropout=0, num_features=cfg.embedding_size, use_se=cfg.SE)
        try:
            # Load pre-trained weights onto the CPU
            backbone.load_state_dict(torch.load(self.model_path, map_location=self.device))
            print("Model loaded successfully.")
        except RuntimeError as e:
            print(f"Error loading model: {e}")
            return None
        backbone.eval()
        return backbone

    def preprocess_image(self, image_path):
        # Preprocess the image
        img = cv2.imread(image_path)
        img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
        img_tensor = self.transform(img_rgb).unsqueeze(0)  # Add batch dimension
        return img_tensor

    def extract_features(self, img_tensor):
        # Extract features using the loaded model
        if self.backbone is None:
            print("Model not loaded.")
            return None
        with torch.no_grad():
            features = F.normalize(self.backbone(img_tensor))
        return features

    def compute_cosine_similarity(self, features_1, features_2):
        # Compute cosine similarity between two sets of features
        return cosine_similarity(features_1.cpu().numpy(), features_2.cpu().numpy())[0][0]

    def run_n_to_n_matching(self, folder_path, output_csv):
        # Process all images in the folder and perform N:N matching
        images = [os.path.join(folder_path, img) for img in os.listdir(folder_path) if img.endswith(('.jpg', '.png'))]

        if len(images) < 2:
            print("Need at least two images for N:N matching.")
            return

        # Extract features for all images
        image_features = {}
        for img_path in images:
            img_tensor = self.preprocess_image(img_path)
            features = self.extract_features(img_tensor)
            image_features[img_path] = features

        # List to store the matching results
        matching_results = []

        # Perform N:N matching by comparing every image pair
        for i, img_path_1 in enumerate(images):
            features_1 = image_features[img_path_1]
            for j, img_path_2 in enumerate(images):
                if i >= j:  # Avoid redundant calculations and self-comparison
                    continue

                features_2 = image_features[img_path_2]
                # Compute cosine similarity between the two images
                similarity = self.compute_cosine_similarity(features_1, features_2)

                # Store the result (image names and similarity score)
                matching_results.append({
                    'image_1': os.path.basename(img_path_1), 
                    'image_2': os.path.basename(img_path_2), 
                    'cosine_similarity': similarity
                })

        # Save matching results to CSV
        df = pd.DataFrame(matching_results)
        df.to_csv(output_csv, index=False)
        print(f"Matching results saved to {output_csv}.")

def main(args):
    model_path = r"example.pth" # Folder containing weights
    folder_path = r"example"  # Folder containing all the images for N:N matching
    output_csv = r"example.csv" Saving csv

    face_recognition_model = FaceRecognitionModel(model_path)
    face_recognition_model.run_n_to_n_matching(folder_path, output_csv)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='N:N matching with cosine similarity')
    parser.add_argument('--local_rank', type=int, default=0, help='local_rank')
    parser.add_argument('--resume', type=int, default=0, help="resume training")
    args_ = parser.parse_args()
    main(args_)

(Insert relevant parts of the code here)

I tested with the following weights:

• CASIA-webface

• SFace-CL

• SFace-CLS

• SFace-KT

Here are the results I observed:

Starting score for non-similar face (mean cosine similarity):

• CASIA-webface: 0.736122

• SFace-CL: 0.75276744

• SFace-CLS: 0.80045915

• SFace-KT: 0.7076743

Final score for similar face (mean cosine similarity):

• CASIA-webface: 0.7416373

• SFace-CL: 0.775683

• SFace-CLS: 0.8008376

• SFace-KT: 0.7437376

Problem: The similarity scores for non-similar faces and similar faces are very close, which limits the accuracy in distinguishing between them. For instance, in the case of SFace-KT, the difference between non-similar (0.7076743) and similar faces (0.7437376) is minimal. This behavior can be seen across all the weights.

Could you provide any suggestions on improving the model's ability to better distinguish between similar and non-similar faces? I suspect there might be an issue in either the weight structure or feature extraction process.

Let me know if you need more information or if there are any suggested tweaks I can make to further improve the results.

[fdbtrs]

Are the images aligned and cropped? you can use this code to align and crop the testing data: https://github.com/fdbtrs/ExFaceGAN/blob/main/MTCNN_alignment.py

[utkarsh-chaurasia]

Yes, I have already cropped the data into the required format. The images are aligned and preprocessed as per the model's requirements.

[fdbtrs]

what are the similar, none-similar faces? Is it the same/different identity? ... P.S. you don't need these lines: transforms.Resize((112, 112)), transforms.RandomHorizontalFlip(),

[utkarsh-chaurasia]

Thank you for your suggestions.

To clarify, by "similar faces," I mean images of the same identity, while "non-similar faces" are images of different identities.

I tried removing the lines:

transforms.Resize((112, 112)),
transforms.RandomHorizontalFlip(),

as you suggested, but it didn’t significantly impact the results. The similarity scores for both similar and non-similar faces remain close, making it difficult to distinguish between them effectively.

What I’m attempting to do is perform N matching between a set of images using cosine similarity based on their feature embeddings. The goal is to see how well the model differentiates between images of the same person and images of different people. However, the similarity scores for non-similar faces (different identities) are still too close to those for similar faces (same identity), which affects the matching accuracy.

Any further suggestions on improving the performance or distinguishing capabilities would be greatly appreciated.