choyingw
commented
1 year ago I’m not sure what inaccurateness you are referring to.
For the groundtruth, the original AFLW2000-3D annotation exactly contains much error. Its annotation process (described in the associated CVPR 2016 paper) is automatic but not manual with several failure cases as you shown here. Thus, there is another reannotated version of AFLW2000-3D for remedy but it’s more recent and the convention is comparison on the original one. I think it makes more sense to evaluate on the reannotated version. Or recently there are some other better-quality datasets for facial alignments.
For the model, I agree that there are still some room of improvement. The model tends to produce large error on highly occluded or very large pose faces, but it’s what we always struggle with.
On Tuesday, June 27, 2023, ken881015 @.***> wrote:
-
Hello, I'm really appreciate the work you completed, SynergyNet is not only light-weighted but also keep an acceptable accuracy on AFLW2000-3D.
Although I'm also one of trainer who can't reproduce NME 3.4% (best is 3.674% after fix code problem in here https://urldefense.com/v3/__https://github.com/choyingw/SynergyNet/issues/18*issuecomment-1600030352__;Iw!!LIr3w8kk_Xxm!pQ5KerkncxONoAPU1G2QmGWgO3mKJwAwO7WOB8Cfuh8ULlea9I9anVPzlt_NWafh8M-lEXbCIaILwF97KqR6cxjX$) on original annotation of benchmark, I keep trying to analysis what kind of images model failed on it and try improve it. So, I sorted NME of 2000 images, make a grid of 48 worst images and the model alignment on it, and show the ground truth of annotation beside it. (for each pair, left is model output and right is ground truth. and its reannotated version). [image: grid_of_worst_alignment_0~47_re_v2_fix_loss_problem_80] https://urldefense.com/v3/__https://user-images.githubusercontent.com/38501223/249102379-3799a5e7-5439-4466-9bf8-73dfd3703a3e.png__;!!LIr3w8kk_Xxm!pQ5KerkncxONoAPU1G2QmGWgO3mKJwAwO7WOB8Cfuh8ULlea9I9anVPzlt_NWafh8M-lEXbCIaILwF97Kpwg4-gH$
As you can see, some annotation of Ground Truth is not accurate, (index start from 1) pair of (1,1) (1,2) is obvious that its annotation is not worth for reference..., but by other pair (e.g. (8,6)), it shows that reason of large NME is due to model performance instead of inaccurate annotation, namely, it's still have chance to be improved.
Here is part of my code to post process the file (roi_box, pts68...) you offer in the repo, and visualize the alignment on image. For the inaccurate problem, did I do anything wrong? or is there any opinion you can share for us? I'll be really appreciated for it.
put this code in ./aflw2000_data/ and you can run it
import matplotlib.pyplot as plt import numpy as np from pathlib import Path
you can select by image name
img_name = "image02156.jpg"
img = plt.imread("./AFLW2000-3D_crop/"+img_name)
choose the version of benchmark annotation (ori or re)
pts68 = np.load("./eval/AFLW2000-3D.pts68.npy") pts68 = np.load("./eval/AFLW2000-3D-Reannotated.pts68.npy")
bbox = np.load("./eval/AFLW2000-3D_crop.roi_box.npy") fname_list = Path("./AFLW2000-3D_crop.list").read_text().strip().split('\n')
coordinate process
pts68[:,0,:] = (pts68[:,0,:] - bbox[:,[0]]) / (bbox[:,[2]] - bbox[:,[0]]) 120 pts68[:,1,:] = (pts68[:,1,:] - bbox[:,[1]]) / (bbox[:,[3]] - bbox[:,[1]]) 120
fig, ax = plt.subplots()
plot image
ax.imshow(img)
scatter landmarks
idx = fname_list.index(img_name) ax.scatter(pts68[idx,0,:], pts68[idx,1,:])
fig.savefig("alignment.jpg")
— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/choyingw/SynergyNet/issues/30__;!!LIr3w8kk_Xxm!pQ5KerkncxONoAPU1G2QmGWgO3mKJwAwO7WOB8Cfuh8ULlea9I9anVPzlt_NWafh8M-lEXbCIaILwF97KiF-N2mM$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AKMLAJI2LHS7D5GFEDLUVQLXNKT7HANCNFSM6AAAAAAZVK3B44__;!!LIr3w8kk_Xxm!pQ5KerkncxONoAPU1G2QmGWgO3mKJwAwO7WOB8Cfuh8ULlea9I9anVPzlt_NWafh8M-lEXbCIaILwF97KtPnkr5j$ . You are receiving this because you are subscribed to this thread.Message ID: @.***>
-- Ph.D. candidate of Computer Science Department University of Southern California
ken881015
Hello, I'm really appreciate the work you completed, SynergyNet is not only light-weighted but also keep an acceptable accuracy on AFLW2000-3D.
Although I'm also one of trainer who can't reproduce NME 3.4% (best is 3.674% after fix code problem in here) on original annotation of benchmark, I keep trying to analysis what kind of images model be failed on it and try improve through training process.
So, I sorted NME of 2000 images, make a grid of 48 worst images and the model alignment on it, and show the ground truth of annotation beside it. (for each pair, left is model output and right is ground truth. and it's reannotated version).
As you can see, some annotation of Ground Truth is not accurate, (index start from 1) pair of (1,1) (1,2) is obvious that its annotation is not worth for reference..., but by other pair (e.g. (8,6)), it shows that reason of large NME is due to model performance instead of inaccurate annotation, namely, it's still have chance to be improved.
Here is part of my code to post process the file (roi_box, pts68...) you offer in the repo, and visualize the alignment on image. For the inaccurate problem, did I do anything wrong? or is there any opinion you can share for us? I'll be really appreciated for it.