[Feature Request] Getting Embeddings For Facial Images with Facial Recognition Models

[serengil]

Thank you for the project!

As I see, you are using pytorch's img2vec to generate embeddings. As an alternative, we may consider checking a face is available in the image, and if yes find its embeddings with a facial recognition model such as FaceNet.

# !pip instal deepface
from deepface import DeepFace

# check face is available in the given image. img can be a file on the filesystem, or numpy array as well.
face_objs = DeepFace.extract_faces(img_path = img, detector_backend="mtcnn")

if faces:
   for face_obj in face_objs 
      detected_face = face_obj["face"]
      embedding = DeepFace.represent(img_path = detected_face, model_name="Facenet", detector_backend="skip")
else:
   # continue to do with pytorch's img2vec

I will be happy to contribute if this attracts your attention. Feel free to close this ticket if you think this will cause losing its way.

[cobanov]

Sounds perfect! Should we implement this as an additional function like def extract_faces() or at the class level? If you have a better suggestion I'd love to hear it.

import importlib

class Tasnif:
    def __init__(self, num_classes, pca_dim=16, use_gpu=False, find_faces=False):
        ...
        ...

        # face detection
        try:
            self.face_module = (
            importlib.import_module("deepface") if self.find_faces else None
        )
        except ImportError:
            raise ValueError(
                "The deepface package is not installed. Please install it with `pip install pip install deepface`"
            )
        ...
        self.find_faces = find_faces
        self.faces = []

    ...

    def calculate(self):
        ...
        if self.find_faces:
            self.faces = deepface.DeepFace.extract_faces(
                img_path=self.images, detector_backend="mtcnn"
            )
            if self.faces:
                for face_obj in self.face_objs 
                    detected_face = face_obj["face"]
                    embedding = DeepFace.represent(img_path = detected_face, model_name="Facenet", detector_backend="skip")
        ...

[serengil]

IMO, we can put it them all into a function like face2vec and call this from get_embeddings

Thinking something like this:

# https://github.com/cobanov/tasnif/blob/main/tasnif/calculations.py

def get_embeddings(use_gpu=False, images=None, find_faces=False):
    """
    This Python function initializes an Img2Vec object, runs it on either GPU or CPU, and retrieves
    image embeddings.
    """

    logging.info(f"Img2Vec is running on {'GPU' if use_gpu else 'CPU'}...")
    img2vec = Img2Vec(cuda=use_gpu)

    embeddings = (
        (find_faces and face2vec(images)) 
        or img2vec.get_vec(images, tensor=False)
        )
    return embeddings

def face2vec(images: List[np.ndarray]) -> List[List[float]]:
    embeddings = []

    try:
        from deepface import DeepFace
    except ImportError:
        raise ValueError(
            "The deepface package is not installed."
            "Please install it with `pip install deepface`"
        )

    for img in images:
        try:
           face_objs = DeepFace.extract_faces(
             img_path = img,
             detector_backend="mtcnn"
           )
           for face_obj in face_objs:
             embedding_obj = DeepFace.represent(
                 img_path = detected_face,
                 model_name="Facenet",
                 detector_backend="skip"
             )
             embedding = embedding_obj[0]["embedding"]
             embeddings.append(embedding)

        except ValueError as err: 
           # in case of no face detected in the given image, ValueError thrown
           # still, facenet can be used to find embeddings
           embedding_obj = DeepFace.represent(
                 img_path = img,
                 model_name="Facenet",
                 detector_backend="skip"
           )
           embedding = embedding_obj[0]["embedding"]
           embeddings.append(embedding)

    return embeddings

# https://github.com/cobanov/tasnif/blob/main/tasnif/tasnif.py
class Tasnif:
    def __init__(self, num_classes, pca_dim=16, use_gpu=False, find_faces=False):
        ...
        ...
        self.find_faces = find_faces
        ...

    def calculate(self):
        self.embeddings = get_embeddings(
            use_gpu=self.use_gpu,
            images=self.images,
            find_faces=find_faces
        )
        ...

[cobanov]

Sefik, this change really excites me, but I will ask for your time until the weekend. I don't want to make a decision without looking carefully.

[serengil]

No problem, take your time please.

Please do not hesitate to contact me if I can give you a hand.

[Stealeristaken]

Is that request still active? I can try to open a pull request for it?

[cobanov]

I was going to look into this issue, but due to a sudden holiday, I still haven't had the opportunity to talk to Sefik @serengil , I apologize for keeping you waiting.

[cobanov]

Hey @serengil , I am very sorry for keeping you waiting for so long, I will review the PR you sent with pleasure at any time you are available.