zhangxuan1918 / InverseRendering3DMMCnn

face fusion project
0 stars 0 forks source link

refactor training script for fine-tuning 3dmm model #11

Closed zhangxuan1918 closed 5 years ago

zhangxuan1918 commented 5 years ago

make it more module

zhangxuan1918 commented 5 years ago

refactor training script

[x] make script module and easy to change [x] remove landmarks header, get landmarks from shape and pose [] do not normalize landmarks when preparing training data [x] make the loss function configurable, e.g. tex loss can be l1 instead of l2

zhangxuan1918 commented 5 years ago

try to write the rendering using tensorflow. Currently the rendering is done using numpy. We need to check if this can be done using tensor, e.g. shape_param is a tensor instead of numpy array. Besides, we need to do it in batch

zhangxuan1918 commented 5 years ago

looks like it's a bit hard to render directly in tensorflow. In https://github.com/tensorflow/graphics/blob/master/tensorflow_graphics/rendering/differentiable_renderer/rasterizer.py, it can render vertices and triangles. But without texture.

Maybe check https://github.com/tranluan/Nonlinear_Face_3DMM/blob/master/rendering_ops.py to see how they do rendering

zhangxuan1918 commented 5 years ago

looks complicated, will turn tensor to numpy and render sequentially

zhangxuan1918 commented 5 years ago

looks complicated, will turn tensor to numpy and render sequentially

cannot do this. If we do this, tensorflow is not able to compute the gradient. Everything has to be done using tensor

  1. checkout py_function in tensorlfow to see if we can turn all the py function to tensorflow. Not sure it will work since we also use cython
  2. either we don't consider landmarks
  3. we rewrite rendering in tensor