Incomplete alpha blend and warping for FaceMorph

[EklavyaFCB]

Hello,

I am trying to morph 2 images from the London Face Dataset. This dataset has usually 189 landmarks included as .tem files, which I'm able to read in readPoints.

However, at the end the final morphed image still has a few black spots on the given face, which also still cropped and surrounded by black rather than the original background.

Can someone tell me where I'm going wrong ? I follow pretty much the same code as FaceMorph.py. Problem seems to come from the length of the calculated Delaunay Triangles, but I do go through all of it's points.

Here are the original 2 images, with and without

--

My modified faceMorph.py:

import numpy as np
import cv2 as cv
import random
import sys

def readPoints(path):
    '''Read points from .tem file'''
    # Create an array of points.
    points = []
    # Read points
    with open(path) as file:
        no_lines = int(file.readline())
        for i, line in enumerate(file):
            if 0 <= i < no_lines:
                x, y = line.split()
                points.append((int(float(x)), int(float(y))))

    return points

def applyAffineTransform(src, srcTri, dstTri, size):
    '''Apply affine transform calculated using srcTri and dstTri to src and output an image of size.'''
    # Given a pair of triangles, find the affine transform.
    warpMat = cv.getAffineTransform(np.float32(srcTri), np.float32(dstTri))

    # Apply the Affine Transform just found to the src image
    dst = cv.warpAffine(src, warpMat, (size[0], size[1]), None,
                        flags=cv.INTER_LINEAR, borderMode=cv.BORDER_REFLECT_101)

    return dst

def morphTriangle(img1, img2, img, t1, t2, t, alpha):
    '''Warps and alpha blends triangular regions from img1 and img2 to img'''
    # Find bounding rectangle for each triangle
    r1 = cv.boundingRect(np.float32([t1]))
    r2 = cv.boundingRect(np.float32([t2]))
    r = cv.boundingRect(np.float32([t]))

    # Offset points by left top corner of the respective rectangles
    t1Rect = []
    t2Rect = []
    tRect = []

    for i in range(0, 3):
        tRect.append(((t[i][0] - r[0]), (t[i][1] - r[1])))
        t1Rect.append(((t1[i][0] - r1[0]), (t1[i][1] - r1[1])))
        t2Rect.append(((t2[i][0] - r2[0]), (t2[i][1] - r2[1])))

    # Get mask by filling triangle
    mask = np.zeros((r[3], r[2], 3), dtype=np.float32)
    cv.fillConvexPoly(mask, np.int32(tRect), (1.0, 1.0, 1.0), 16, 0)

    # Apply warpImage to small rectangular patches
    img1Rect = img1[r1[1]:r1[1] + r1[3], r1[0]:r1[0] + r1[2]]
    img2Rect = img2[r2[1]:r2[1] + r2[3], r2[0]:r2[0] + r2[2]]

    size = (r[2], r[3])
    warpImage1 = applyAffineTransform(img1Rect, t1Rect, tRect, size)
    warpImage2 = applyAffineTransform(img2Rect, t2Rect, tRect, size)

    # Alpha blend rectangular patches
    imgRect = (1.0 - alpha) * warpImage1 + alpha * warpImage2

    # Copy triangular region of the rectangular patch to the output image
    img[r[1]:r[1]+r[3], r[0]:r[0]+r[2]] = img[r[1]:r[1] + r[3], r[0]:r[0]+r[2]] * (1 - mask) + imgRect * mask

def rect_contains(rect, point):
    '''Check if a point is inside a rectangle'''
    if point[0] < rect[0]:
        return False
    elif point[1] < rect[1]:
        return False
    elif point[0] > rect[2]:
        return False
    elif point[1] > rect[3]:
        return False
    return True

def draw_point(img, p, color):
    '''Draw a point'''
    cv.circle(img, p, 2, color, cv.FILLED, cv.LINE_AA, 0)

def draw_voronoi(img, subdiv):
    '''Draw voronoi diagram'''
    (facets, centers) = subdiv.getVoronoiFacetList([])

    for i in range(0, len(facets)):
        ifacet_arr = []
        for f in facets[i]:
            ifacet_arr.append(f)

        ifacet = np.array(ifacet_arr, np.int)
        color = (random.randint(0, 255), random.randint(
            0, 255), random.randint(0, 255))

        cv.fillConvexPoly(img, ifacet, color, cv.LINE_AA, 0)
        ifacets = np.array([ifacet])
        cv.polylines(img, ifacets, True, (0, 0, 0), 1, cv.LINE_AA, 0)
        cv.circle(img, (centers[i][0], centers[i][1]),
                  3, (0, 0, 0), cv.FILLED, cv.LINE_AA, 0)

def draw_delaunay(img, subdiv, delaunay_color):
    '''Draw delaunay triangles'''
    triangleList = subdiv.getTriangleList()
    size = img.shape
    r = (0, 0, size[1], size[0])

    for t in triangleList:
        pt1 = (t[0], t[1])
        pt2 = (t[2], t[3])
        pt3 = (t[4], t[5])

        if rect_contains(r, pt1) and rect_contains(r, pt2) and rect_contains(r, pt3):
            cv.line(img, pt1, pt2, delaunay_color, 1, cv.LINE_AA, 0)
            cv.line(img, pt2, pt3, delaunay_color, 1, cv.LINE_AA, 0)
            cv.line(img, pt3, pt1, delaunay_color, 1, cv.LINE_AA, 0)

def calculateDelaunayTriangles(rect, subdiv, points, img, win_delaunay, delaunay_color, draw=False):
    '''Calculate delanauy triangle'''

    # Insert points into subdiv
    for p in points:
        subdiv.insert((p[0], p[1]))

    # List of triangles. Each triangle is a list of 3 points ( 6 numbers )
    triangleList = subdiv.getTriangleList()

    # Find the indices of triangles in the points array
    delaunayTri = []

    for t in triangleList:
        pt = []
        pt.append((t[0], t[1]))
        pt.append((t[2], t[3]))
        pt.append((t[4], t[5]))

        pt1 = (t[0], t[1])
        pt2 = (t[2], t[3])
        pt3 = (t[4], t[5])

        if rect_contains(rect, pt1) and rect_contains(rect, pt2) and rect_contains(rect, pt3):
            ind = []
            for j in range(0, 3):
                for k in range(0, len(points)):
                    if(abs(pt[j][0] - points[k][0]) < 1.0 and abs(pt[j][1] - points[k][1]) < 1.0):
                        ind.append(k)
            if len(ind) == 3:
                delaunayTri.append((ind[0], ind[1], ind[2]))

            # Draw lines
            if draw:
                cv.line(img, pt1, pt2, delaunay_color, 1, cv.LINE_AA, 0)
                cv.line(img, pt2, pt3, delaunay_color, 1, cv.LINE_AA, 0)
                cv.line(img, pt3, pt1, delaunay_color, 1, cv.LINE_AA, 0)
                imgS = cv.resize(img, (413, 531))

    return delaunayTri

def main():
        # Variables
    filename1 = '../../../Data/Facelab_London/neutral_front/Raw/007_03.jpg'
    filename2 = '../../../Data/Facelab_London/neutral_front/Raw/009_03.jpg'
    alpha = 0.5

    # Define window names
    win_delaunay = 'Delaunay Triangulation'
    win_voronoi  = 'Voronoi Diagram'

    # Define colors for drawing.
    delaunay_color = (255, 255, 255)
    points_color   = (0, 0, 255)

    # Read images
    img1 = cv.imread(filename1)
    img2 = cv.imread(filename2)

    img1_copy = cv.imread(filename1)
    img2_copy = cv.imread(filename2)

    # Convert Mat to float data type
    # img1   = np.float32(img1)
    # img2   = np.float32(img2)

    # img1_copy = np.float32(img1_copy)
    # img2_copy = np.float32(img2_copy)

    # Read array of corresponding points
    points1 = readPoints(filename1[:-4] + '.tem')
    points2 = readPoints(filename2[:-4] + '.tem')
    points = []

    # Compute weighted average point coordinates
    for i in range(0, len(points1)):
        x = (1 - alpha) * points1[i][0] + alpha * points2[i][0]
        y = (1 - alpha) * points1[i][1] + alpha * points2[i][1]
        points.append((x, y))

    # Allocate space for final output
    imgMorph = np.zeros(img1.shape,   dtype=img1.dtype)

    # Rectangle to be used with Subdiv2D
    size = img1.shape
    rect = (0, 0, size[1], size[0])

    # Create an instance of Subdiv2D
    subdiv = cv.Subdiv2D(rect)

    # Calculate and draw delaunay triangles
    delaunayTri = calculateDelaunayTriangles(rect, subdiv, points1, img1_copy, win_delaunay,
                                             (255, 255, 255), draw=True)
    _ = calculateDelaunayTriangles(rect, subdiv, points2, img2_copy, win_delaunay,
                                   (255, 255, 255), draw=True)

    # Allocate space for Voronoi Diagram
    img_voronoi = np.zeros(img1.shape,   dtype=img1.dtype)

    # Draw Voronoi diagram
    draw_voronoi(img_voronoi,  subdiv)

    # Print
    print("delaunayTri:", len(delaunayTri))
    print("points: ", len(points))
    print("points1:", len(points1))
    print("points2:", len(points2))

    # Morph by reading calculated triangles
    for line in delaunayTri:
        x, y, z = line

        x = int(x)
        y = int(y)
        z = int(z)

        t1 = [points1[x], points1[y], points1[z]]
        t2 = [points2[x], points2[y], points2[z]]
        t = [points[x],  points[y],  points[z]]

        # Morph one triangle at a time.
        morphTriangle(img1, img2, imgMorph, t1, t2, t, alpha)

    # Resize images
    desired_size = (860, 860)
    img1_copyS   = cv.resize(img1_copy, desired_size)
    img2_copyS   = cv.resize(img2_copy, desired_size)
    img1S        = cv.resize(img1, desired_size)
    img2S        = cv.resize(img2, desired_size)
    img_voronoiS = cv.resize(img_voronoi, desired_size)
    imgMorphS    = cv.resize(imgMorph, desired_size)

    # Save Images
    cv.imwrite("Image_1.jpg",        img1S)
    cv.imwrite("Image_2.jpg",        img2S)
    cv.imwrite("Image_1_copy.jpg",   img1_copyS)
    cv.imwrite("Image_2_copy.jpg",   img2_copyS)
    # cv.imwrite("Win_Voronoi.jpg",    img_voronoiS)
    cv.imwrite("Morphed_Face.jpg",   imgMorphS)

if __name__ == "__main__":
    main()

[EklavyaFCB]

It seems the triangulation on the first image was correct, but not the second one .

By creating a new rect2 and subdiv2 to parse into calculateDelaunayTriangles(rect2, subdiv2, points2, img2_copy, win_delaunay, (255, 255, 255), draw=True), I was able to get the correct triangulation on the 2nd image.

Now the list returned by calculateDelaunayTriangles with both sets of inputs (rect & subdiv, and rect2 & subdiv2) and the list returned by subdiv.getTriangleList() seems to be both different:

triangleList 353
triangleList 359
delaunayTri 1: 331
delaunayTri 2: 351

The final morphed image is thus still incomplete:

[lipi17dpatnaik]

Hi

Can you provide the corresponding tem files so that we can replicate and try to resolve the issue?

[EklavyaFCB]

Sure, here you go:

Tem.zip

EDIT: added both the .tem and the .jpg files.

[EklavyaFCB]

I mean in the code I've given above, I should be calling calculateDelaunayTriangles with points, which contains the average mean of points1 and points2, rather than on these two.

Nonetheless, even when calling it on points, I get an incomplete morph ... Modified the delaunayTri part, see:

def main():
    # Variables
    filename1 = '../../../Data/Facelab_London/neutral_front/Raw/007_03.jpg'
    filename2 = '../../../Data/Facelab_London/neutral_front/Raw/009_03.jpg'
    alpha = 0.5

    # Define window names
    win_delaunay = 'Delaunay Triangulation'
    win_voronoi = 'Voronoi Diagram'

    # Define colors for drawing.
    d_col = (255, 255, 255)
    p_col = (0, 0, 255)

    # Read images
    img1 = cv.imread(filename1)
    img2 = cv.imread(filename2)

    img1_copy = cv.imread(filename1)
    img2_copy = cv.imread(filename2)

    # Read array of corresponding points
    points1 = readPoints(filename1[:-4] + '.tem')
    points2 = readPoints(filename2[:-4] + '.tem')
    points = []

    # Compute weighted average point coordinates
    for i in range(0, len(points1)):
        x = (1 - alpha) * points1[i][0] + alpha * points2[i][0]
        y = (1 - alpha) * points1[i][1] + alpha * points2[i][1]
        points.append((x, y))

    # Allocate space for final output
    imgMorph = np.zeros(img1.shape, dtype=img1.dtype)

    # Rectangle to be used with Subdiv2D
    size = img1.shape
    rect = (0, 0, size[1],  size[0])

    # Create an instance of Subdiv2D
    subdiv = cv.Subdiv2D(rect)

    # Calculate and draw delaunay triangles
    delaunayTri = calculateDelaunayTriangles(rect, subdiv, points, img1_copy, win_delaunay, d_col, draw=False)

    # Allocate space for Voronoi Diagram
    img_voronoi = np.zeros(img1.shape, dtype=img1.dtype)

    # Draw Voronoi diagram
    draw_voronoi(img_voronoi, subdiv)

    # Morph by reading calculated triangles
    for line in delaunayTri:
        x, y, z = line

        x = int(x)
        y = int(y)
        z = int(z)

        t1 = [points1[x], points1[y], points1[z]]
        t2 = [points2[x], points2[y], points2[z]]
        t  = [points[x],  points[y],  points[z]]

        # Morph one triangle at a time.
        morphTriangle(img1, img2, imgMorph, t1, t2, t, alpha)

    # Resize images
    desired_size = (860, 860)
    img1_copyS   = cv.resize(img1_copy,   desired_size)
    img2_copyS   = cv.resize(img2_copy,   desired_size)
    img1S        = cv.resize(img1,        desired_size)
    img2S        = cv.resize(img2,        desired_size)
    img_voronoiS = cv.resize(img_voronoi, desired_size)
    imgMorphS    = cv.resize(imgMorph,    desired_size)

    # Save Images
    cv.imwrite("Image_1.jpg", img1S)
    cv.imwrite("Image_2.jpg", img2S)
    cv.imwrite("Image_1_copy.jpg", img1_copyS)
    cv.imwrite("Image_2_copy.jpg", img2_copyS)
    cv.imwrite("Morphed_Face.jpg", imgMorphS)

[lipi17dpatnaik]

Can you also share your complete Python code for face morphing that you are using?

[EklavyaFCB]

Hi, sorry was different versions. Here is the definitive one.

I just execute the code below (you'll just have to change the filename1 and filename2 variable paths:

import numpy as np
import cv2 as cv
import random
import sys

def readPoints(path):
    '''Read points from .tem file'''
    # Create an array of points.
    points = []
    # Read points
    with open(path) as file:
        no_lines = int(file.readline())
        for i, line in enumerate(file):
            if 0 <= i < no_lines:
                x, y = line.split()
                points.append((int(float(x)), int(float(y))))

    return points

def applyAffineTransform(src, srcTri, dstTri, size):
    '''Apply affine transform calculated using srcTri and dstTri to src and output an image of size.'''
    # Given a pair of triangles, find the affine transform.
    warpMat = cv.getAffineTransform(np.float32(srcTri), np.float32(dstTri))

    # Apply the Affine Transform just found to the src image
    dst = cv.warpAffine(src, warpMat, (size[0], size[1]), None,
                        flags=cv.INTER_LINEAR, borderMode=cv.BORDER_REFLECT_101)

    return dst

def morphTriangle(img1, img2, img, t1, t2, t, alpha):
    '''Warps and alpha blends triangular regions from img1 and img2 to img'''
    # Find bounding rectangle for each triangle
    r1 = cv.boundingRect(np.float32([t1]))
    r2 = cv.boundingRect(np.float32([t2]))
    r = cv.boundingRect(np.float32([t]))

    # Offset points by left top corner of the respective rectangles
    t1Rect = []
    t2Rect = []
    tRect = []

    for i in range(0, 3):
        tRect.append(((t[i][0] - r[0]), (t[i][1] - r[1])))
        t1Rect.append(((t1[i][0] - r1[0]), (t1[i][1] - r1[1])))
        t2Rect.append(((t2[i][0] - r2[0]), (t2[i][1] - r2[1])))

    # Get mask by filling triangle
    mask = np.zeros((r[3], r[2], 3), dtype=np.float32)
    cv.fillConvexPoly(mask, np.int32(tRect), (1.0, 1.0, 1.0), 16, 0)

    # Apply warpImage to small rectangular patches
    img1Rect = img1[r1[1]:r1[1] + r1[3], r1[0]:r1[0] + r1[2]]
    img2Rect = img2[r2[1]:r2[1] + r2[3], r2[0]:r2[0] + r2[2]]

    size = (r[2], r[3])
    warpImage1 = applyAffineTransform(img1Rect, t1Rect, tRect, size)
    warpImage2 = applyAffineTransform(img2Rect, t2Rect, tRect, size)

    # Alpha blend rectangular patches
    imgRect = (1.0 - alpha) * warpImage1 + alpha * warpImage2

    # Copy triangular region of the rectangular patch to the output image
    img[r[1]:r[1]+r[3], r[0]:r[0]+r[2]] = img[r[1]:r[1] +
                                              r[3], r[0]:r[0]+r[2]] * (1 - mask) + imgRect * mask

def rect_contains(rect, point):
    '''Check if a point is inside a rectangle'''
    if point[0] < rect[0]:
        return False
    elif point[1] < rect[1]:
        return False
    elif point[0] > rect[2]:
        return False
    elif point[1] > rect[3]:
        return False
    return True

def draw_point(img, p, color):
    '''Draw a point'''
    cv.circle(img, p, 2, color, cv.FILLED, cv.LINE_AA, 0)

def draw_voronoi(img, subdiv):
    '''Draw voronoi diagram'''
    (facets, centers) = subdiv.getVoronoiFacetList([])

    for i in range(0, len(facets)):
        ifacet_arr = []
        for f in facets[i]:
            ifacet_arr.append(f)

        ifacet = np.array(ifacet_arr, np.int)
        color = (random.randint(0, 255), random.randint(
            0, 255), random.randint(0, 255))

        cv.fillConvexPoly(img, ifacet, color, cv.LINE_AA, 0)
        ifacets = np.array([ifacet])
        cv.polylines(img, ifacets, True, (0, 0, 0), 1, cv.LINE_AA, 0)
        cv.circle(img, (centers[i][0], centers[i][1]),
                  3, (0, 0, 0), cv.FILLED, cv.LINE_AA, 0)

def draw_delaunay(img, subdiv, delaunay_color):
    '''Draw delaunay triangles'''
    triangleList = subdiv.getTriangleList()
    size = img.shape
    r = (0, 0, size[1], size[0])

    for t in triangleList:
        pt1 = (t[0], t[1])
        pt2 = (t[2], t[3])
        pt3 = (t[4], t[5])

        if rect_contains(r, pt1) and rect_contains(r, pt2) and rect_contains(r, pt3):
            cv.line(img, pt1, pt2, delaunay_color, 1, cv.LINE_AA, 0)
            cv.line(img, pt2, pt3, delaunay_color, 1, cv.LINE_AA, 0)
            cv.line(img, pt3, pt1, delaunay_color, 1, cv.LINE_AA, 0)

def calculateDelaunayTriangles(rect, subdiv, points, img, win_delaunay, delaunay_color, draw=False):
    '''Calculate delanauy triangle'''

    # Insert points into subdiv
    for p in points:
        subdiv.insert((p[0], p[1]))

    # List of triangles. Each triangle is a list of 3 points (6 numbers)
    triangleList = subdiv.getTriangleList()
    print('triangleList', len(triangleList))

    # Find the indices of triangles in the points array
    delaunayTri = []

    for t in triangleList:
        pt = []
        pt.append((t[0], t[1]))
        pt.append((t[2], t[3]))
        pt.append((t[4], t[5]))

        pt1 = (t[0], t[1])
        pt2 = (t[2], t[3])
        pt3 = (t[4], t[5])

        if rect_contains(rect, pt1) and rect_contains(rect, pt2) and rect_contains(rect, pt3):
            ind = []
            for j in range(0, 3):
                for k in range(0, len(points)):
                    if(abs(pt[j][0] - points[k][0]) < 1.0 and abs(pt[j][1] - points[k][1]) < 1.0):
                        ind.append(k)

            if len(ind) == 3:
                delaunayTri.append((ind[0], ind[1], ind[2]))

            # Draw lines
            if draw:
                cv.line(img, pt1, pt2, delaunay_color, 1, cv.LINE_AA, 0)
                cv.line(img, pt2, pt3, delaunay_color, 1, cv.LINE_AA, 0)
                cv.line(img, pt3, pt1, delaunay_color, 1, cv.LINE_AA, 0)
                imgS = cv.resize(img, (413, 531))

    return delaunayTri

def main():
    # Variables
    filename1 = '../../../Data/Facelab_London/neutral_front/Raw/007_03.jpg'
    filename2 = '../../../Data/Facelab_London/neutral_front/Raw/009_03.jpg'
    alpha = 0.5

    # Define window names
    win_delaunay = 'Delaunay Triangulation'
    win_voronoi = 'Voronoi Diagram'

    # Define colors for drawing.
    d_col = (255, 255, 255)
    p_col = (0, 0, 255)

    # Read images
    img1 = cv.imread(filename1)
    img2 = cv.imread(filename2)

    img1_copy = cv.imread(filename1)
    img2_copy = cv.imread(filename2)

    # Read array of corresponding points
    points1 = readPoints(filename1[:-4] + '.tem')
    points2 = readPoints(filename2[:-4] + '.tem')
    points = []

    # Compute weighted average point coordinates
    for i in range(0, len(points1)):
        x = (1 - alpha) * points1[i][0] + alpha * points2[i][0]
        y = (1 - alpha) * points1[i][1] + alpha * points2[i][1]
        points.append((x, y))

    # Allocate space for final output
    imgMorph = np.zeros(img1.shape, dtype=img1.dtype)

    # Rectangle to be used with Subdiv2D
    size = img1.shape
    rect = (0, 0, size[1],  size[0])

    # Create an instance of Subdiv2D
    subdiv = cv.Subdiv2D(rect)

    # Calculate and draw delaunay triangles
    delaunayTri = calculateDelaunayTriangles(rect, subdiv, points, img1_copy, win_delaunay, d_col, draw=False)

    # Allocate space for Voronoi Diagram
    img_voronoi = np.zeros(img1.shape, dtype=img1.dtype)

    # Draw Voronoi diagram
    draw_voronoi(img_voronoi, subdiv)

    # Morph by reading calculated triangles
    for line in delaunayTri:
        x, y, z = line

        x = int(x)
        y = int(y)
        z = int(z)

        t1 = [points1[x], points1[y], points1[z]]
        t2 = [points2[x], points2[y], points2[z]]
        t  = [points[x],  points[y],  points[z]]

        # Morph one triangle at a time.
        morphTriangle(img1, img2, imgMorph, t1, t2, t, alpha)

    # Resize images
    desired_size = (860, 860)
    img1_copyS   = cv.resize(img1_copy,   desired_size)
    img2_copyS   = cv.resize(img2_copy,   desired_size)
    img1S        = cv.resize(img1,        desired_size)
    img2S        = cv.resize(img2,        desired_size)
    img_voronoiS = cv.resize(img_voronoi, desired_size)
    imgMorphS    = cv.resize(imgMorph,    desired_size)

    # Save Images
    cv.imwrite("Image_1.jpg", img1S)
    cv.imwrite("Image_2.jpg", img2S)
    cv.imwrite("Image_1_copy.jpg", img1_copyS)
    cv.imwrite("Image_2_copy.jpg", img2_copyS)
    cv.imwrite("Morphed_Face.jpg", imgMorphS)

if __name__ == "__main__":
    main()

[lipi17dpatnaik]

Thanks! I will look into it and get back to you.

[EklavyaFCB]

Code.zip

As a .zip

[EklavyaFCB]

Any updates ? :slightly_smiling_face:

[lipi17dpatnaik]

Changing alpha to 0.8, and the if condition in the calculateDelaunayTriangles function to if(abs(pt[j][0] - points[k][0]) < 0.1 and abs(pt[j][1] - points[k][1]) < 0.5) improves the result slightly. The black spot in the lip is still present. Still working on it.

[EklavyaFCB]

Sure, thanks for letting me know. Perhaps one should try extracting the facial landmarks through dlib and see if that works for some reason. I don't know if the black spots seem are related to the facial landmarks - the code does read in 189 of them from the .tem files, but doesn't seem to be able to complete the morph on any 2 pairs of image I give it from this dataset.

[lipi17dpatnaik]

Hi @EklavyaFCB that's correct. After replacing your points with the facial landmarks obtained using 68 point model, the results obtained are perfectly correct. So the issue is not with your code, but it is actually with the landmark points. Please find attached the image obtained after landmark detection.

[EklavyaFCB]

I see, thank you. Perhaps the additional number of facial landmarks conflicts with triangulation in some way ... Could you kindly send the code for extracting the dlib landmarks please ?

Also, how would one then put this face back onto the original image, i.e. to obtain the final image (with background, neck, etc) ?

[EklavyaFCB]

@lipi17dpatnaik I get the landmarks points from dlib but still get the black spot :sleepy:. What value did you use for alpha?

Can you please share you code ?

Here is my modified code:

def main():
    # Variables
    filename1 = '../../../Data/Facelab_London/neutral_front/Raw/007_03.jpg'
    filename2 = '../../../Data/Facelab_London/neutral_front/Raw/009_03.jpg'
    alpha = 0.5

    # Define window names
    win_delaunay = 'Delaunay Triangulation'
    win_voronoi = 'Voronoi Diagram'

    # Define colors for drawing.
    d_col = (255, 255, 255)
    p_col = (0, 0, 255)

    # Read images
    img1 = cv.imread(filename1)
    img2 = cv.imread(filename2)

    img1_copy = cv.imread(filename1)
    img2_copy = cv.imread(filename2)

    # Read array of corresponding points
    #points1 = readPoints(filename1[:-4] + '.tem')
    #points2 = readPoints(filename2[:-4] + '.tem')

    # Read points using dlib
    detector = dlib.get_frontal_face_detector()
    predictor = dlib.shape_predictor('shape_predictor_68_face_landmarks.dat')

    rects1 = detector(img1, 1)
    rects2 = detector(img2, 1)

    points1 = predictor(img1, rects1[0])
    points2 = predictor(img2, rects2[0])

    points1 = face_utils.shape_to_np(points1)
    points2 = face_utils.shape_to_np(points2)

    points = []

    # Compute weighted average point coordinates
    for i in range(0, len(points1)):
        x = (1 - alpha) * points1[i][0] + alpha * points2[i][0]
        y = (1 - alpha) * points1[i][1] + alpha * points2[i][1]
        points.append((x, y))

    # Draw points
    (x, y, w, h) = face_utils.rect_to_bb(rects1[0])
    for (x, y) in points1:
        cv.circle(img1, (x, y), 2, p_col, -1)

    (x, y, w, h) = face_utils.rect_to_bb(rects2[0])
    for (x, y) in points2:
        cv.circle(img2, (x, y), 2, p_col, -1)

    # Allocate space for final output
    imgMorph = np.zeros(img1_copy.shape, dtype=img1_copy.dtype)

    # Rectangle to be used with Subdiv2D
    size = img1_copy.shape
    rect = (0, 0, size[1],  size[0])

    # Create an instance of Subdiv2D
    subdiv = cv.Subdiv2D(rect)

    # Calculate and draw delaunay triangles
    delaunayTri = calculateDelaunayTriangles(rect, subdiv, points, img1_copy, win_delaunay, d_col, draw=False)

    # Allocate space for Voronoi Diagram
    img_voronoi = np.zeros(img1_copy.shape, dtype=img1_copy.dtype)

    # Draw Voronoi diagram
    draw_voronoi(img_voronoi, subdiv)

    # Morph by reading calculated triangles
    for line in delaunayTri:
        x, y, z = line

        x = int(x)
        y = int(y)
        z = int(z)

        t1 = [points1[x], points1[y], points1[z]]
        t2 = [points2[x], points2[y], points2[z]]
        t  = [points[x],  points[y],  points[z]]

        # Morph one triangle at a time.
        morphTriangle(img1_copy, img2_copy, imgMorph, t1, t2, t, alpha)

    # Resize images
    desired_size = (860, 860)
    img1_copyS   = cv.resize(img1_copy,   desired_size)
    img2_copyS   = cv.resize(img2_copy,   desired_size)
    img1S        = cv.resize(img1,        desired_size)
    img2S        = cv.resize(img2,        desired_size)
    img_voronoiS = cv.resize(img_voronoi, desired_size)
    imgMorphS    = cv.resize(imgMorph,    desired_size)

    # Save Images
    cv.imwrite("Image_1.jpg", img1S)
    cv.imwrite("Image_2.jpg", img2S)
    cv.imwrite("Image_1_copy.jpg", img1_copyS)
    cv.imwrite("Image_2_copy.jpg", img2_copyS)
    cv.imwrite("Morphed_Face.jpg", imgMorphS)

[EklavyaFCB]

Ah, changing the if condition in the calculateDelaunayTriangles function to if(abs(pt[j][0] - points[k][0]) < 0.1 and abs(pt[j][1] - points[k][1]) < 0.5) like you'd mentioned earlier fixed it !

I don't get what these parameters represent.

[lipi17dpatnaik]

@EklavyaFCB please find attached the revised Python code. Change the landmark model path as required. OpenCV_Morph.zip

[EklavyaFCB]

@lipi17dpatnaik Thank you, I'm able to run it and get the same results as mine.

Do you know how to complete the morph i.e. have the face on rest of the body and not on a black background, as given on examples on this post: https://www.learnopencv.com/face-morph-using-opencv-cpp-python/

[lipi17dpatnaik]

You can use a simple binary masking technique so that wherever there is 0 (black) in the new image, you can directly take the corresponding pixel from the original image (on which you want to morph). Think of it like a bitwise operation.

[EklavyaFCB]

Ugh, the black lip spot is still present when I run the algorithm on different images. It's hard to compile a statistic but it seems like every 2nd picture has the spot.

I tried changing the value in the if-statement in calculateDelaunayTriangles but it makes spots appear on another picture if it fixes the current one ...

[EklavyaFCB]

@lipi17dpatnaik Hey, could you kindly tell me which version of dlib you were using ?

[lipi17dpatnaik]

@EklavyaFCB 19.19.0

[kelseyo430]

@EklavyaFCB I realize I am a year late to this thread, but where you ever able to resolve this issue? I have the same problem and the masking technique suggested seems like a bit of a cop-out. I appreciate any insight!

[EklavyaFCB]

Nope, I was never able to solve the issue despite spending 2 weeks on it. The solution proposed above removed the spots from some images but not all. It's some dumb issue.

[xbnz-1997]

Hello, I am trying to morph 2 pictures from my own, I read the blog postFace Morph Using OpenCV — C++ / Python. Howerver, I don't know how to get the tri.txt by performing Delaunay Triangulation. I appreciated for your help.

[brmarkus]

Your referenced article says these:

But it is very easy to find a few point correspondences. For morphing two dissimilar objects, like a cat’s face and a human’s face, we can click on a few points on the two images to establish correspondences and interpolate the results for the rest of the pixels.

As well as using Facial-Landmark-detection:

I used dlib to detect 68 corresponding points. Next, I added four more points ( one on the right hand side ear, one on the neck, and two on the shoulders ). Finally, I added the corners of the image and half way points between those corners as corresponding points as well. Needless to say, one can add a few more points around the head and neck to get even better results, or remove the manually clicked points to get slightly worse ( but fully automatic ) results.

And then, referring to the Delaunay Triangulation article, you call subdiv.getTriangleList() and store the list in the file. (or keep it in memory and continue with the morphing).