Retrieve identity 3D

[HassanLH]

Hi, first of all thank you for sharing this work

I am trying to retrieve the 3D model. I have to use the .shp and .expression files which contain the 3DMM parameters and use the 3DMM basis to compute the 3D but I am having trouble finding which file stores it.

Is there code to do that in the project that I may not have found please ? Otherwise, which file contains what I would need to compute the 3D model

[sariyanidi]

Hi -- sorry for the delay here, I was on vacation but happy to address this issue!

The code by default stores the 3D mesh only for the neutral, expressionless face. If you use the default configuration file, this mesh should be found under the folder output/BFMmm-19830.cfg1.global4. For example, if you run the demo video elaine.mp4, the neutral 3D mesh should be written to the file output/BFMmm-19830.cfg1.global4/elaine.shp.

If you need the 3D mesh with expression variation, then something like the code below should work (I modified the code from the save_identity_and_shape function in the file process_video.py but have not tested it, minor errors may occur).

def save_3d_mesh_with_identity_and_expression(alpha, eps, shp_path, morphable_model="BFMmm-19830"):
    # alpha: identity (shape) coefficients
    # exp: expression coefficients for one frame (should be a vector with 79 entries for the default 3DMM)
    # shp_path: location where the mesh will be stored
    # morphable_model: string that specifies the morphable model
    sdir = './models/MMs/%s' % morphable_model 
    IX  = np.loadtxt('%s/IX.dat' % sdir)
    IY  = np.loadtxt('%s/IY.dat' % sdir)
    IZ  = np.loadtxt('%s/IZ.dat' % sdir)

    # The following is the expression basis of the 3DMM 
    EX  = np.loadtxt('%s/E/EX_79.dat' % sdir)
    EY  = np.loadtxt('%s/E/EY_79.dat' % sdir)
    EZ  = np.loadtxt('%s/E/EZ_79.dat' % sdir)

    x0 = np.loadtxt('%s/X0_mean.dat' % sdir)
    y0 = np.loadtxt('%s/Y0_mean.dat' % sdir)
    z0 = np.loadtxt('%s/Z0_mean.dat' % sdir)

    x = (x0+(IX @ alpha) + (EX @ eps)).reshape(-1,1)
    y = (y0+(IY @ alpha) + (EY @ eps)).reshape(-1,1)
    z = (z0+(IZ @ alpha) + (EZ @ eps)).reshape(-1,1)

    np.savetxt(shp_path, np.concatenate((x,y,z), axis=1))

Happy to clarify further or write a more complete piece of code if this is still needed.