Some faces with one eye hair covered can't be recognized

[modelsex]

First THANKS A LOT for all contributors' hard work! Always make a compare test after big change, test with same source 1000 pics (kar801 -> kar1800) , compare with FakeApp1.1 & latest faceswap commit 232d931. *Test files [Link Removed]

Expected behavior

Not sure, limitation ? or possible to improve ?

Actual behavior

FakeApp1.1 extract rate is 988/1000 faceswap -D cnn extract rate is 943/1000

[Image Removed]

Notice that some faces - specially one eye covered by hair can't be extract. Example: kar1086 -> kar1090, these 5 pics can be extract normally in FakeApp, but failed in faceswap. Compare kar1085 with kar1086, no big gap in these 2 pics, just corner of the eye be covered by hair in kar1086.  

Steps to reproduce

python faceswap.py extract -i D:/project4/data_A1/ -o D:/project4/data_A1/output/ -D cnn

Other relevant information

• Operating system and version: Windows, macOS, Linux Windows10 Python3.6.4 CUDA9.0 dlib 19.9 The others env same as requirements-gpu-python36-cuda9.txt

[oatssss]

From what commit did you do the extraction from?

[modelsex]

Latest commit 232d931

[iperov]

this is dlib cnn issue anybody know how to decompile fakeapp v1.1?

[Kirin-kun]

There's a program "a.exe" in the alignment subdirectory of FakeApp 1.1

C:\FakeApp\alignment>a.exe -h RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb usage: a.exe [-h] [--one-face] [--all-faces] [--startFrame STARTFRAME] [--maxFrames MAXFRAMES] [--fileType FILETYPE] [--processor PROCESSOR] input_dir [output_dir] [output_file]

positional arguments: input_dir output_dir output_file

optional arguments: -h, --help show this help message and exit --one-face --all-faces --startFrame STARTFRAME --maxFrames MAXFRAMES --fileType FILETYPE --processor PROCESSOR

It seems to be a compiled python exe. I guess someone could attempt to decompile it using unpy2exe to extract the .pyc and then use pyREtic to get the source code.

[iperov]

unpy2exe not works with a.exe

[iperov]

kar1086.png - just minimal program with dlib.cnn_face_detection_model_v1 doesnt detect any face rectangles. So issue not relate to last commits

[iperov]

face_recognition also find nothing

frame = cv2.imread(sys.argv[1])
s = face_recognition.face_locations (frame, model='cnn')

FakeApp has DLIB version 19.8.1N, but I cannot install this version on Windows and check.

[Kirin-kun]

I tested various scenarios by dumping 696 pictures in a directory, with lots of different angles and faces of different people and I tried all 3 face detectors.

FakeApp (1.1 and 2.2) both detected 645 faces (including the false positives). Most of them good, very few of them a misdection.

faceswap HOG detected only 370 (including the false positive)! Obviously, HOG is only able to detect faces when they are looking at the camera and upright.

faceswap cnn (latest commit) detected 610 (including the false positive), which is better, but missed some that FakeApp didn't. It improved a lot.

faceswapp cnn seems to not be able to detect faces seen from the bottom, but is better at detecting small faces than FakeApp. It seems to also have problems detecting the face when it's next to an angle of the image or something.

Here are two pictures (mostly SFW) that fakeapp detects correctly and that faceswap cnn does not:

[Images Removed]

When I cropped the first one to keep the face in the middle of the picture, cnn suddenly detected it!

FakeApp must have a different training base or something.

[Jack29913]

@Kirin-kun could you try using this https://github.com/1adrianb/face-alignment

[iperov]

@Kirin-kun faceswapp cnn seems to not be able to detect faces and again, this is not faceswap cnn. This is DLIB CNN, which 100% same at fakeapp, faceswap, face_recognition, face_alignment.

[iperov]

mmod_human_face_detector.dat - is base for dlib cnn 100% identical

[Kirin-kun]

@iperov I successfully decompiled the a.exe from FakeApp 1.1.

Using https://sourceforge.net/projects/pyinstallerextractor to extract the exe. It produces a bunch of pyc.

Then "pip install uncompyle6" on windows to see what's in them. Obviously, FakeApp uses face-alignment from 1adrianb.

C:\FakeApp\alignment\a.exe_extracted\out00-PYZ.pyz_extracted>uncompyle6 face_alignment.pyc

# uncompyle6 version 3.0.0
# Python bytecode 3.6 (3379)
# Decompiled from: Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1
900 64 bit (AMD64)]
# Embedded file name: face_alignment\__init__.py
__author__ = 'Adrian Bulat'
__email__ = 'adrian.bulat@nottingham.ac.uk'
__version__ = '0.1.0'
from .api import FaceAlignment, LandmarksType, NetworkSize
# okay decompiling face_alignment.pyc

[iperov]

there is no useful info

[Jack29913]

@Kirin-kun Hmm I think it's safe to say that face-alignment/1adrianb finds more faces at least for some scenarios. It's possible that faceapp did some modifications on it. To confirm you can try the images directly on the library I shared the link above.

[Kirin-kun]

face_alignment.api.py seems to be the relevant file.

` from future import print_function import os, glob, dlib, torch, torch.nn as nn from torch.autograd import Variable from enum import Enum try: import urllib.request as request_file except BaseException: import urllib as request_file

from .models import FAN, ResNetDepth from .utils import * from settings import processor

class LandmarksType(Enum): _2D = 1 _2halfD = 2 _3D = 3

class NetworkSize(Enum): LARGE = 4

def __new__(cls, value):
    member = object.__new__(cls)
    member._value_ = value
    return member

def __int__(self):
    return self.value

class FaceAlignment: """Initialize the face alignment pipeline

Args:
    landmarks_type (``LandmarksType`` object): an enum defining the type of predicted points.
    network_size (``NetworkSize`` object): an enum defining the size of the network (for the 2D and 2.5D points).
    enable_cuda (bool, optional): If True, all the computations will be done on a CUDA-enabled GPU (recommended).
    enable_cudnn (bool, optional): If True, cudnn library will be used in the benchmark mode
    flip_input (bool, optional): Increase the network accuracy by doing a second forward passed with
                                the flipped version of the image
    use_cnn_face_detector (bool, optional): If True, dlib's CNN based face detector is used even if CUDA
                                            is disabled.

Example:
    >>> FaceAlignment(NetworkSize.2D, flip_input=False)
"""

def __init__(self, landmarks_type, network_size=NetworkSize.LARGE, enable_cuda=True, enable_cudnn=True, flip_input=False, use_cnn_face_detector=False):
    if processor == 'CPU':
        self.enable_cuda = False
    else:
        self.enable_cuda = True
    self.use_cnn_face_detector = use_cnn_face_detector
    self.flip_input = flip_input
    self.landmarks_type = landmarks_type
    base_path = os.path.join(appdata_dir('face_alignment'), 'data')
    if not os.path.exists(base_path):
        os.makedirs(base_path)
    if enable_cudnn:
        if self.enable_cuda:
            torch.backends.cudnn.benchmark = True
    if self.enable_cuda or self.use_cnn_face_detector:
        path_to_detector = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'mmod_human_face_detector.dat')
        self.face_detector = dlib.cnn_face_detection_model_v1(path_to_detector)
    else:
        self.face_detector = dlib.get_frontal_face_detector()
    self.face_alignemnt_net = FAN(int(network_size))
    if landmarks_type == LandmarksType._2D:
        network_name = '2DFAN-' + str(int(network_size)) + '.pth.tar'
    else:
        network_name = '3DFAN-' + str(int(network_size)) + '.pth.tar'
    fan_path = os.path.join(os.path.dirname(os.path.dirname(__file__)), network_name)
    fan_weights = (torch.load)(fan_path, map_location=(lambda storage, loc: storage))
    fan_dict = {k.replace('module.', ''):v for k, v in fan_weights['state_dict'].items()}
    self.face_alignemnt_net.load_state_dict(fan_dict)
    if self.enable_cuda:
        self.face_alignemnt_net.cuda()
    self.face_alignemnt_net.eval()
    if landmarks_type == LandmarksType._3D:
        self.depth_prediciton_net = ResNetDepth()
        depth_model_path = os.path.join(base_path, 'depth.pth.tar')
        if not os.path.isfile(depth_model_path):
            print('Downloading the Face Alignment depth Network (FAN-D). Please wait...')
            request_file.urlretrieve('https://www.adrianbulat.com/downloads/python-fan/depth.pth.tar', os.path.join(depth_model_path))
        depth_weights = (torch.load)(depth_model_path, map_location=(lambda storage, loc: storage))
        depth_dict = {k.replace('module.', ''):v for k, v in depth_weights['state_dict'].items()}
        self.depth_prediciton_net.load_state_dict(depth_dict)
        if self.enable_cuda:
            self.depth_prediciton_net.cuda()
        self.depth_prediciton_net.eval()

def detect_faces(self, image):
    """Run the dlib face detector over an image

    Args:
        image (``ndarray`` object or string): either the path to the image or an image previosly opened
        on which face detection will be performed.

    Returns:
        Returns a list of detected faces
    """
    return self.face_detector(image, 1)

def get_landmarks(self, input_image, all_faces=False):
    if isinstance(input_image, str):
        try:
            image = io.imread(input_image)
        except IOError:
            print('error opening file :: ', input_image)
            return

    else:
        image = input_image
    detected_faces = self.detect_faces(image)
    if len(detected_faces) > 0:
        landmarks = []
        for i, d in enumerate(detected_faces):
            if i > 1:
                if not all_faces:
                    break
            if self.enable_cuda or self.use_cnn_face_detector:
                d = d.rect
            center = torch.FloatTensor([
             d.right() - (d.right() - d.left()) / 2.0,
             d.bottom() - (d.bottom() - d.top()) / 2.0])
            center[1] = center[1] - (d.bottom() - d.top()) * 0.1
            scale = (d.right() - d.left() + d.bottom() - d.top()) / 200.0
            inp = crop(image, center, scale)
            inp = torch.from_numpy(inp.transpose((2, 0, 1))).float().div(255.0).unsqueeze_(0)
            if self.enable_cuda:
                inp = inp.cuda()
            out = self.face_alignemnt_net(Variable(inp, volatile=True))[-1].data.cpu()
            if self.flip_input:
                out += flip((self.face_alignemnt_net(Variable((flip(inp)), volatile=True))[-1].data.cpu()), is_label=True)
            pts, pts_img = get_preds_fromhm(out, center, scale)
            pts, pts_img = pts.view(68, 2) * 4, pts_img.view(68, 2)
            if self.landmarks_type == LandmarksType._3D:
                heatmaps = np.zeros((68, 256, 256))
                for i in range(68):
                    if pts[(i, 0)] > 0:
                        heatmaps[i] = draw_gaussian(heatmaps[i], pts[i], 2)

                heatmaps = torch.from_numpy(heatmaps).view(1, 68, 256, 256).float()
                if self.enable_cuda:
                    heatmaps = heatmaps.cuda()
                depth_pred = self.depth_prediciton_net(Variable((torch.cat((
                 inp, heatmaps), 1)), volatile=True)).data.cpu().view(68, 1)
                pts_img = torch.cat((
                 pts_img, depth_pred * (1.0 / (256.0 / (200.0 * scale)))), 1)
            landmarks.append(pts_img.numpy())

    else:
        print('Warning: No faces were detected.')
        return
    return landmarks

def process_folder(self, path, all_faces=False):
    types = ('*.jpg', '*.png')
    images_list = []
    for files in types:
        images_list.extend(glob.glob(files))

    predictions = []
    for image_name in images_list:
        predictions.append(image_name, self.get_landmarks(image_name, all_faces))

    return predictions`

[iperov]

I built dlib for VS2015 and ran example dnn_mmod_face_detection_ex.cpp and it found face ! even without pyramid_up() upsampling

Same dlib built for python found no faces :-\

[iperov]

I found bug. LOOOL.

[iperov]

[Image Removed]

[iperov]

https://github.com/deepfakes/faceswap/pull/236

[modelsex]

Amazing !!! Will do more test later, thanks !!!!!

[Clorr]

@modelsex just a thought btw: if your faces are too similar, your training may overfit. Please keep in mind it is better to extract a frame only each 2 to 5s

[iperov]

@Clorr not necessarily overfit. For example FakeApp result https://www.youtube.com/watch?v=Va9JLpkCUBs I trained it with every frame of video.

[deepfakesclub]

I'm getting one-third the frame extraction rates with the latest commit compared to the original face_recognition. (not counting the loading time) Is anyone else experiencing this?

In terms of face extraction quality, I would say the current commit is probably the best general one. It is the most sensitive face extractor, although the error rate can be high at times. I'm posting the numbers I get over in the playground repo.

I do notice that the different implementations have errors on different frames at times, so it's nice having multiple implementations to backup each other for problematic frames.

I still support keep more than one implementation (hog is just plain bad, doesn't count) around if possible.

Posting my numbers/benchmarks here: https://github.com/deepfakes/faceswap-playground/issues/81