rawmarshmellows / FacialClusteringPipeline

28 stars 9 forks source link

An attempt at an end to end pipeline for facial clustering

Introduction

This is an end to end pipeline for facial clustering of photos (similar to what Apple and Android has). Please look at pipeline/Facial Clustering.ipynb to see the current progress.

Method

There are three main steps in creating an end to end pipeline for the facial clustering of images:

  1. Detecting and creating the bounding boxes for the faces

    • For detecting and creating the bounding boxes for the faces I'm using the Multi-Task Cascaded Convolutional Networks ( MTCNN) which has 3 CNNs with the output of one network feeding into the next. Each step refines the bounding boxes for the faces, with the first network being looser on the conditions for the bounding boxes and the last network being the most strict.
  2. Creating the embeddings for the faces

    • For creating the embeddings I'm using a modified version FaceNet which is a Resnet-Inception model that creates a 128 dimensional embedding for a face.
  3. Clustering the faces

    • For clustering the faces I'm using the Rank-Order clustering algorithm. This is a kind of agglomerative clustering technique, which merges the embeddings based on the rank-order distance, and a cluster-level normalized distance.

As I was unable to train the models myself David Sandberg has graciously trained MTCNN and Resnet-Inception models and I was able to modify his code to my use case. Here is a link to the Github page

To setup the environment and install required packages

Assuming that you are using the Anaconda package manager.

  1. Download/Clone the repo and cd to it
  2. Create a new virtual environment: conda create -n facialClustering python=3
  3. Install Tensorflow: pip install tensorflow
  4. Download the weights of Resnet-Inception from here and put it in the pipeline directory. I have already included two folders of faces which are subsets from the Labeled Faces in the Wild dataset.
  5. Go into the pipeline folder and open Jupyter Notebook: jupyter notebook
  6. Open Facial Clustering.ipynb
  7. Hopefully you will be able to open it, if it doesn't work you can download the Facial+Clustering.html and read the notebook.

Results

alt text

TODO:

Done: