AlbedoMM Clipping Textures

[natanielruiz]

Hi (cc @waps101),

When I use the AlbedoMM texture model I get texture clipping like this:

Notice the eyebrows in the sample above.

Another example with a rendered face:

A third example with less clipping:

[TimoBolkart]

Can you please specify how you use the AlbedoMM texture model? To you clip the texture values before exporting?

[natanielruiz]

I am using the sample_texture.py script without modification to save the textures. The texture components are randomly sampled. For the last image though, all components are 0 except the first two which are -2 (2 std below the mean). I believe this shouldn't be too far outside the model's capability but it seems like it is? Or is this an issue that happens because the model has been adapted to work with FLAME. I don't know if this is an issue with the original texture model implemented in Scala.

[TimoBolkart]

This is weird, I am running sample_texture.py with the first two parameters assigned to -2. All with the albedoModel2020_FLAME_albedoPart.npz albedo model. The resulting texture map looks like this:

However that does not seem to have that clipping artifacts. Can you point out, what is different to your experiment? More specifically, I run following code:

with tf.Session() as session:
        session.run(tf.global_variables_initializer())
        tex_params = np.zeros(num_tex_pc)
        tex_params[:2] = -2
        print(tex_params)

        assign_tex = tf.assign(tf_tex_params, tex_params[np.newaxis,:])
        session.run([assign_tex])

        v, tex = session.run([tf_model, tf_tex])
        out_mesh = Mesh(v, smpl.f)
        out_mesh.vt = texture_model['vt']
        out_mesh.ft = texture_model['ft']

        out_mesh_fname = os.path.join(out_path, 'tex_sample_-2.obj')

        out_tex_fname = out_mesh_fname.replace('obj', 'png')
        cv2.imwrite(out_tex_fname, tex)
        out_mesh.set_texture_image(out_tex_fname)
        out_mesh.write_obj(out_mesh_fname)```

[natanielruiz]

Sorry I meant to say -3 instead of -2. Here is the sample I get:

My code is the same as yours except -3 instead of -2

[natanielruiz]

Please let me know if you've had any luck reproducing this. Thank you!

[TimoBolkart]

Unfortunately, I cannot reproduce your result. Running with -3 I get this:

Even when running with -10, it does not show such clipping artifacts

Running with -10 certainly does not make sense but still no weird clipping artifacts occur.

Can you please provide 1) exactly the code that you run that outputs your result? 2) the list of packages installed in your virtual environment, as shown by running pip list in your terminal?

[natanielruiz]

Okay this is great then! It means there is something going wrong on my end and that the model is good. I will investigate and then update this. For now let me close it. Thanks so much.

[natanielruiz]

@TimoBolkart Okay. I am reopening. I re-cloned the project and used a clean conda venv to reinstall all packages and then re-ran the code. I get the same result.

The code

'''
Max-Planck-Gesellschaft zur Foerderung der Wissenschaften e.V. (MPG) is holder of all proprietary rights on this
computer program.

You can only use this computer program if you have closed a license agreement with MPG or you get the right to use
the computer program from someone who is authorized to grant you that right.

Any use of the computer program without a valid license is prohibited and liable to prosecution.

Copyright 2019 Max-Planck-Gesellschaft zur Foerderung der Wissenschaften e.V. (MPG). acting on behalf of its
Max Planck Institute for Intelligent Systems and the Max Planck Institute for Biological Cybernetics.
All rights reserved.

More information about FLAME is available at http://flame.is.tue.mpg.de.
For comments or questions, please email us at flame@tue.mpg.de
'''

import os
import cv2
import six
import argparse
import numpy as np
import tensorflow as tf
from psbody.mesh import Mesh
from psbody.mesh.meshviewer import MeshViewer
from utils.landmarks import load_binary_pickle, load_embedding, tf_get_model_lmks, create_lmk_spheres

from tf_smpl.batch_smpl import SMPL
from tensorflow.contrib.opt import ScipyOptimizerInterface as scipy_pt

def sample_texture(model_fname, texture_fname, num_samples, out_path):
    '''
    Sample the FLAME model to demonstrate how to vary the model parameters.FLAME has parameters to
        - model identity-dependent shape variations (paramters: shape),
        - articulation of neck (paramters: pose[0:3]), jaw (paramters: pose[3:6]), and eyeballs (paramters: pose[6:12])
        - model facial expressions, i.e. all expression motion that does not involve opening the mouth (paramters: exp)
        - global translation (paramters: trans)
        - global rotation (paramters: rot)
    :param model_fname          saved FLAME model
    :param num_samples          number of samples
    :param out_path             output path to save the generated templates (no templates are saved if path is empty)
    '''

    tf_trans = tf.Variable(np.zeros((1,3)), name="trans", dtype=tf.float64, trainable=True)
    tf_rot = tf.Variable(np.zeros((1,3)), name="pose", dtype=tf.float64, trainable=True)
    tf_pose = tf.Variable(np.zeros((1,12)), name="pose", dtype=tf.float64, trainable=True)
    tf_shape = tf.Variable(np.zeros((1,300)), name="shape", dtype=tf.float64, trainable=True)
    tf_exp = tf.Variable(np.zeros((1,100)), name="expression", dtype=tf.float64, trainable=True)
    smpl = SMPL(model_fname)
    tf_model = tf.squeeze(smpl(tf_trans,
                               tf.concat((tf_shape, tf_exp), axis=-1),
                               tf.concat((tf_rot, tf_pose), axis=-1)))

    texture_model = np.load(texture_fname)
    if ('MU' in texture_model) and ('PC' in texture_model) and ('specMU' in texture_model) and ('specPC' in texture_model):
        b_albedoMM = True
    elif ('mean' in texture_model) and ('tex_dir' in texture_model):
        b_albedoMM = False
    else:
        print('Unknown texture model - %s' % texture_fname)
        return

    if b_albedoMM:
        # Albedo Morphable Model 
        num_tex_pc = texture_model['PC'].shape[-1]
        tex_shape = texture_model['MU'].shape

        tf_tex_params = tf.Variable(np.zeros((1,num_tex_pc)), name="params", dtype=tf.float64, trainable=True)

        tf_MU = tf.Variable(np.reshape(texture_model['MU'], (1,-1)), name='MU', dtype=tf.float64, trainable=False)
        tf_PC = tf.Variable(np.reshape(texture_model['PC'], (-1, num_tex_pc)).T, name='PC', dtype=tf.float64, trainable=False)
        tf_specMU = tf.Variable(np.reshape(texture_model['specMU'], (1,-1)), name='specMU', dtype=tf.float64, trainable=False)
        tf_specPC = tf.Variable(np.reshape(texture_model['specPC'], (-1, num_tex_pc)).T, name='specPC', dtype=tf.float64, trainable=False)

        tf_diff_albedo = tf.add(tf_MU, tf.matmul(tf_tex_params, tf_PC))
        tf_spec_albedo = tf.add(tf_specMU, tf.matmul(tf_tex_params, tf_specPC))
        tf_tex = 255*tf.math.pow(0.6*tf.add(tf_diff_albedo, tf_spec_albedo), 1.0/2.2)
    else:
        # MPI texture space or equivalent
        num_tex_pc = texture_model['tex_dir'].shape[-1]
        tex_shape = texture_model['mean'].shape

        tf_tex_params = tf.Variable(np.zeros((1,num_tex_pc)), name="params", dtype=tf.float64, trainable=True)
        tf_tex_mean = tf.Variable(np.reshape(texture_model['mean'], (1,-1)), name='tex_mean', dtype=tf.float64, trainable=False)
        tf_tex_dir = tf.Variable(np.reshape(texture_model['tex_dir'], (-1, num_tex_pc)).T, name='tex_dir', dtype=tf.float64, trainable=False)
        tf_tex = tf.add(tf_tex_mean, tf.matmul(tf_tex_params, tf_tex_dir))

    tf_tex = tf.reshape(tf_tex, (tex_shape[0], tex_shape[1], tex_shape[2]))
    tf_tex = tf.cast(tf.clip_by_value(tf_tex, 0.0, 255.0), tf.int64)

    with tf.Session() as session:
        session.run(tf.global_variables_initializer())
        tex_params = np.zeros(num_tex_pc)
        tex_params[:2] = -3
        print(tex_params)

        assign_tex = tf.assign(tf_tex_params, tex_params[np.newaxis,:])
        session.run([assign_tex])

        v, tex = session.run([tf_model, tf_tex])
        out_mesh = Mesh(v, smpl.f)
        out_mesh.vt = texture_model['vt']
        out_mesh.ft = texture_model['ft']

        out_mesh_fname = os.path.join(out_path, 'tex_sample.obj')
        out_tex_fname = out_mesh_fname.replace('obj', 'png')
        cv2.imwrite(out_tex_fname, tex)
        out_mesh.set_texture_image(out_tex_fname)
        out_mesh.write_obj(out_mesh_fname)

def main(args):
    if not os.path.exists(args.model_fname):
        print('FLAME model not found - %s' % args.model_fname)
        return
    if not os.path.exists(args.texture_fname):
        print('Texture model not found - %s' % args.texture_fname)
        return
    if not os.path.exists(args.out_path):
        os.makedirs(args.out_path)
    sample_texture(args.model_fname, args.texture_fname, int(args.num_samples), args.out_path)

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Sample FLAME shape space')
    parser.add_argument('--model_fname', default='./models/generic_model.pkl', help='Path of the FLAME model')
    parser.add_argument('--texture_fname', default='./models/FLAME_texture.npz', help='Path of the texture model')
    parser.add_argument('--num_samples', default='5', help='Number of samples')
    parser.add_argument('--out_path', default='./texture_samples', help='Output path')
    args = parser.parse_args()
    main(args)

The code is almost exactly the code you have (and that is in the repo).

For pip list

Package              Version
-------------------- ---------
absl-py              0.12.0
astor                0.8.1
backcall             0.2.0
cached-property      1.5.2
certifi              2020.12.5
chumpy               0.70
cycler               0.10.0
decorator            4.4.2
freetype-py          2.2.0
future               0.18.2
gast                 0.2.2
google-pasta         0.2.0
grpcio               1.36.1
h5py                 3.1.0
imageio              2.9.0
importlib-metadata   3.7.2
ipython              7.16.1
ipython-genutils     0.2.0
jedi                 0.18.0
Keras-Applications   1.0.8
Keras-Preprocessing  1.1.2
kiwisolver           1.3.1
Markdown             3.3.4
matplotlib           3.3.4
networkx             2.2
numpy                1.19.5
opencv-python        4.5.1.48
opt-einsum           3.3.0
parso                0.8.1
pexpect              4.8.0
pickleshare          0.7.5
Pillow               8.1.2
pip                  21.0.1
prompt-toolkit       3.0.16
protobuf             3.15.5
psbody-mesh          0.4
ptyprocess           0.7.0
pyglet               1.4.0b1
Pygments             2.8.1
PyOpenGL             3.1.5
pyparsing            2.4.7
pyrender             0.1.33
python-dateutil      2.8.1
PyYAML               5.4.1
pyzmq                22.0.3
scipy                1.5.4
setuptools           54.1.1
six                  1.15.0
tensorboard          1.15.0
tensorflow-estimator 1.15.1
tensorflow-gpu       1.15.2
termcolor            1.1.0
traitlets            4.3.3
trimesh              3.5.15
typing-extensions    3.7.4.3
wcwidth              0.2.5
Werkzeug             1.0.1
wheel                0.36.2
wrapt                1.12.1
zipp                 3.4.1

I followed installation instructions precisely, and I re-downloaded both the face and texture models. The only differences I could think of my setup with the default setup is that I am using CentOS and the fact that I installed chumpy 0.70 instead of 0.69 because pip cannot find that version.

(OS info)

LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.9.2009 (Core)
Release:        7.9.2009
Codename:       Core

Thank you for your help.

[natanielruiz]

I have found the line of code that creates the clipping artifacts:

tf_tex = tf.cast(tf.clip_by_value(tf_tex, 0.0, 255.0), tf.int64)

Do you have that same line in your code? If I comment it out then I have the following result for -3.

Solved!

[TimoBolkart]

Thank you for the feedback. I do have that line in my code and and it was actually there to prevent such clipping artifacts, by mapping too small or too large values to 0 / 255. I don't understand right now why that solves your problem.

[natanielruiz]

Very strange. Thank you for your help though.