Suggestion: 3D Face Reconstruction via Direct Volumetric CNN Regression

[frosty3907]

ala: http://cvl-demos.cs.nott.ac.uk/vrn/

This achieves good results from a single image, imagine what could be done with larger datasets.

Could be used for missing angles.

[kellurian]

Its not a bad idea, getting the OBJ files to either JPG or PNG without uploading would be the trick. In addition, the "faces" it produces do seem rather limited in size. I have thought about this for 3d printing several times.

[frosty3907]

the site itself has a renderer - not that hard to dumps some pngs from there, and in terms of limited in size- well we're only looking at 128x128px

[kellurian]

The object files aren't too detailed- I looked at them in simplify3d to see about printing some and they are kinda messy. Still having trouble getting any .png files from them. It's not a a to b conversion, and the site doesn't have, that I can find, a spot to output .png files from the obj files. Let me know if you see a solution.

[ruah1984]

https://github.com/AaronJackson/vrn.

[iperov]

how it can help with faceswapping ?

[ruah1984]

@iperov, this just help to simulate different angle of face, base on 3D face reconstruction. but i think unless we can know which angle we don't have, if not we actually create more dataset for each 2D face.

[Jack29913]

There's this script in dfaker's repo - I saw this earlier somewhere else- Maybe we can get missing positions using this script.

https://github.com/dfaker/df/blob/master/imageGapsScanner.py

[ruah1984]

As i know the imageGapsScanner may require both.json files for data A and B.

[kcimit]

@ruah1984, like everything else in @dfaker project, I find it very relevant.

I also tried this reconstruction project - using output results from 1 photo does not gives useful results, Some face features are simply missing - like using frontal photo, you will never get nostrils, chin etc.

There are some commercial software which can do some part of the job - facegen or facial studio which can reconstruct from 2 photos and apply morphing.

[deepfakesclub]

This issue would perhaps be better off in the playground repo.

Anyways, I tried playing around with this tool a long time ago, as well as a bunch of other software packages for 3D face reconstruction. For the .obj files, I think MeshLab was the free viewer I used that supports textures.

The bottom line is that I don't think the quality is good enough to use except for very specialized and lucky/random cases. I see great results for 3DMM in papers, but when I try out various published code, the results look like crap unless you carefully choose your examples. They don't seem very robust.

Also, none of these tools are quite "programmatic" enough. The github ones are all a pain to build or run (proprietary expensive matlab...). The free software packages I tried were very hands-on. Both the free and commercial packages also looked too "cartoony" to me.

If you are doing manual work, there are probably better methods for data augmentation. For example, you can find similar faces with the required pose. You need to carefully blend the faces, kind of like a DJ, so you don't overpower the desired face B. You can also do the classic - train both autoencoders on face A, then switch one to face B. That is easy, but I personally hate it, as the new face always looks too contaminated by face A.

The most promising 3D reconstruction tools usually didn't release their code or looked way too complicated for me to try. Some of the more sophisticated ones like the GANs mentioned by Clorr or somebody elsewhere might work.

[dfaker]

I don't think you'd need to generate full new voxel clouds just to generate new angles we don't care about a useful model, just new angles to enrich the samples.

It'd be possible to take an average 3d face (a simple 3d model), rotate it, then project the 2d face points onto it face-aligned noting the X and Y positions from the non-font-aligned viewing angle, that'd give you two sets of points F being a 2d font-on points from he original image and P the transformed positions of where that face point would be if you rotated the head, feed F and P into cv2.remap to get a transformed face and paste that back over the original background.

Pre-calculation of a set of F and P pairs would make this very fast.

[babilio]

Yeah, I've tried the facial reconstruction from the link and it doesn't really reconstruct the face properly, like you would get a nose or chin that are too big or small and it ends up with model results that don't look like the person

[deepfakesclub]

I think this paper (pdf link) is basically doing a variant of what @dfaker suggests. It uses generic 3D models to rapidly construct alternative angle views.

I believe there is a github with code for this, or a similar one. Couldn't find it at the moment. The issue was that it only generates 5-6 fixed angles of rotation (along yaw), and they didn't release the full generic 3D models for generating arbitrary angles.

[Clorr]

@deepfakesclub interesting paper you suggest here. I was thinking about that one but it only generates frontal view which is not really the hardest to find usually. However the approach was interesting and the 2 pathway decoder was also somewhat interesting

[torzdf]

Just cleaning up issues. I'm not sure this is yet in a position where it can be used in this project, but please feel free to request to reopen if you believe otherwise.