search

AaronJackson / vrn

:man: Code for "Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression"
http://aaronsplace.co.uk/papers/jackson2017recon/
MIT License
4.52k stars 746 forks source link

about Voxelisation #92

Closed marvin521 closed 6 years ago

marvin521 commented 6 years ago

I wanna know whether the volumetric data is binary or not? It seems like that the volumetric data should be binary because of Eq (1) (i.e., cross entropy loss). However, If the volumetric data is binary, the net tends to segment the face instead of regressing the face shape.

AaronJackson commented 6 years ago

A threshold is applied during training (although it works just as well to train non-binary with NME, for example). During inference there the values are not binary because this stops the output looking blocky (given the right isosurface implementation)

marvin521 commented 6 years ago

Thanks for quickly response. Could you tell me the hole filling algorithm you used in the process of voxelisation? Thanks!!!

AaronJackson commented 6 years ago

We rotated the mesh to frontal with the provided pose parameters, then we produced a depth map from this by interpolating over the front most points. This might be a bit problematic on detail-heavy datasets, but 3DDFA models are so smooth anyway, it doesn't really matter. Once that's done, we rotate the volume back with the same pose parameters so that the voxelised face is in alignment with the image again,

marvin521 commented 6 years ago

I don't know how to rotate the voxels back to the image space after voxelising frontal face meshes. The main reason is that the image has been scaled while the vertices of the mesh is still in the original image space. How embarrassed it is~~~ The process of re-implementing the training code is like to find One Piece.