Some mistakes hit me when I was implementing the image fitting procedure and texture extraction procedure following the official demo.
I found that the rendered result which is shown side by side with the original image is not aligned with the original face. And also the extracted texture based on this fitted result is incorrect. And I will show some modifications here to help someone facing the same problem as me.
To begin with, my corresponding Python package settings are:
Python 3.9.18
dlib 19.24.2
Descriptions:
The result I get when processing chan.jpg following official demo is:
landmark:fitting result:texture:
It is interesting that is model seems to fit the face in the image well, but why is the rendered result not aligned with the face?
I reviewed the official code and found that there are many Y-axis flipping operations in detecting landmarks and extracting texture. Such as some codes in facescape_fitter.py:
Also, in some cases, the cropped image will have a zero-width or zero-height in get_texture, so we can add a judgment before using the img1Cropped, to make the code more robust:
# Apply warpImage to small rectangular patches
img1Cropped = img[croppedRectangleInImage[0]:(croppedRectangleInImage[0] + croppedRectangleInImage[2]), croppedRectangleInImage[1]:(croppedRectangleInImage[1] + croppedRectangleInImage[3])]
# ADD THIS: Check if image is empty, in some cases the cropped image may be empty
if img1Cropped.shape[0] == 0 or img1Cropped.shape[1] == 0:
continue
warpMat = cv2.getAffineTransform(np.float32(triangleVertexPixelCoordInCroppedImage), np.float32(triangleVertexPixelCoordInCroppedTexture))
# Get mask by filling triangle
...
After modifications:
landmark detecting:texture extraction:
For aligning the rendered result with the face in the original image, we only need to add a Y-axis flipping to mesh vertices:
Some mistakes hit me when I was implementing the
image fitting
procedure andtexture extraction
procedure following the official demo. I found that the rendered result which is shown side by side with the original image is not aligned with the original face. And also the extracted texture based on this fitted result is incorrect. And I will show some modifications here to help someone facing the same problem as me.To begin with, my corresponding Python package settings are:
Python 3.9.18
dlib 19.24.2
Descriptions:
The result I get when processing
fitting result:
texture:
![chan_result_face_alignment_texture](https://github.com/zhuhao-nju/facescape/assets/131634363/60d7dbd1-fc1f-46db-8a38-5e402495c073)
chan.jpg
following official demo is: landmark:It is interesting that is model seems to fit the face in the image well, but why is the rendered result not aligned with the face?
I reviewed the official code and found that there are many Y-axis flipping operations in detecting landmarks and extracting texture. Such as some codes in
facescape_fitter.py
:And,
These operations seem to make the fitted mesh result more compatible with
pyrender
process. But in my case, it leads to an incorrect result.Solutions
I made some modifications, which cancel the Y-axis flipping:
Also, in some cases, the cropped image will have a zero-width or zero-height in
get_texture
, so we can add a judgment before using theimg1Cropped
, to make the code more robust:After modifications: landmark detecting:
texture extraction:
![chan_result_face_alignment_texture_correct](https://github.com/zhuhao-nju/facescape/assets/131634363/7166396e-ded1-448e-81a1-c861805e7e59)
For aligning the rendered result with the face in the original image, we only need to add a Y-axis flipping to mesh vertices:
Now the alignment is correct:![correct-alignment](https://github.com/zhuhao-nju/facescape/assets/131634363/f1713b5d-21e1-4eb5-87b3-cdd926570051)