problems on facescape data

[WXizhi]

Hello, dear authors, I've read your paper and downloaded the data, great job! I run the demo with Facescape publishable data. I fed in the _fittingpointclouds.py with mvs point clould that has been aligned with nphm coordinate. The neutral expession reuslt looks good yet other expressions seem to be wrong.... I followed the steps described in seciton Fitting Point Clouds , and used the your pretrained data. Could you please help me figure out the problem? thank you very much!

facescpae_344_6_jaw_right.zip

[SimonGiebenhain]

Hey,

Can you provide more details about how you aligned the FaceScape scan into the NPHM coordinate system? I suspect that this is the cause for the bad performance (Note that the reconstructed expressions often have some small flaws in the mouth area for NPHM. But I guess it should be that bad.)

It appears to me that the alignment is not perfect. E.g. when comparing to examples from the dataset looks like this

While the difference is subtle (you can notice it, especially for the chin and forehead), I guess it confuses the model quite a bit. Since the registration is quite consistent in the dataset this is a definitely a small outlier.

One suggestion would be to fit FLAME to the model first, since it shares the same coordinate system with NPHM (you just have to scale with a factor 4 from FLAME to NPHM coordinate system).

Kind Regards, Simon

[WXizhi]

Hey,

Can you provide more details about how you aligned the FaceScape scan into the NPHM coordinate system? I suspect that this is the cause for the bad performance (Note that the reconstructed expressions often have some small flaws in the mouth area for NPHM. But I guess it should be that bad.)

It appears to me that the alignment is not perfect. E.g. when comparing to examples from the dataset looks like this

While the difference is subtle (you can notice it, especially for the chin and forehead), I guess it confuses the model quite a bit. Since the registration is quite consistent in the dataset this is a definitely a small outlier.

One suggestion would be to fit FLAME to the model first, since it shares the same coordinate system with NPHM (you just have to scale with a factor 4 from FLAME to NPHM coordinate system).

Kind Regards, Simon

Thank you for your advice, I'll fit FLAME first!

[Zvyozdo4ka]

@WXizhi Did you fit FLAME?

[Zvyozdo4ka]

Hello, dear authors, I've read your paper and downloaded the data, great job! I run the demo with Facescape publishable data. I fed in the _fittingpointclouds.py with mvs point clould that has been aligned with nphm coordinate. The neutral expession reuslt looks good yet other expressions seem to be wrong.... I followed the steps described in seciton Fitting Point Clouds , and used the your pretrained data.

Could you please explain how did you get these results? Fitting Point Clouds comprises only two lines of explanations and it did not work for me. Could you explain please?

[WXizhi]

Hello, dear authors, I've read your paper and downloaded the data, great job! I run the demo with Facescape publishable data. I fed in the _fittingpointclouds.py with mvs point clould that has been aligned with nphm coordinate. The neutral expession reuslt looks good yet other expressions seem to be wrong.... I followed the steps described in seciton Fitting Point Clouds , and used the your pretrained data.

Could you please explain how did you get these results? Fitting Point Clouds comprises only two lines of explanations and it did not work for me. Could you explain please?

Hello, I get the mvs point cloud with COLMAP. Yet a rigid transformation(trans/rot/scale) is needed to get the point cloud aligned with the FLAME coordinate. By now, I manually select landmark pairs on FLAME(or other 3DMM) mesh and the mvs point cloud, then run Procrustes algorithm to get the rigid transformation. It is a bit lengthy, if you have better ways, please let me know.

[Zvyozdo4ka]

Hello, I get the mvs point cloud with COLMAP. Yet a rigid transformation(trans/rot/scale) is needed to get the point cloud aligned with the FLAME coordinate. By now, I manually select landmark pairs on FLAME(or other 3DMM) mesh and the mvs point cloud, then run Procrustes algorithm to get the rigid transformation. It is a bit lengthy, if you have better ways, please let me know.

Hello, @WXizhi . I got point cloud in colmap, i got intrinsic and extrinsic parameters from Colmap

How did you align landmark pairs on FLAME mesh and point cloud? Using some script or app?

I fed in the _fittingpointclouds.py with mvs point clould that has been aligned with nphm coordinate.

how to test, just use point cloud without aligning, just to fit point clouds. I saw that you mentioned above that you 'fed in the _fittingpointclouds.py', and how did you actually fed your point cloud in _fittingpointclouds.py ?

[WXizhi]

Hello, I get the mvs point cloud with COLMAP. Yet a rigid transformation(trans/rot/scale) is needed to get the point cloud aligned with the FLAME coordinate. By now, I manually select landmark pairs on FLAME(or other 3DMM) mesh and the mvs point cloud, then run Procrustes algorithm to get the rigid transformation. It is a bit lengthy, if you have better ways, please let me know.

Hello, @WXizhi . I got point cloud in colmap, i got intrinsic and extrinsic parameters from Colmap

How did you align landmark pairs on FLAME mesh and point cloud? Using some script or app?

I fed in the _fittingpointclouds.py with mvs point clould that has been aligned with nphm coordinate.

how to test, just use point cloud without aligning, just to fit point clouds. I saw that you mentioned above that you 'fed in the _fittingpointclouds.py', and how did you actually fed your point cloud in _fittingpointclouds.py ?

1. How did you align landmark pairs on FLAME mesh and point cloud? I manually select the landmark pairs, and run Procrustes analysis(https://en.wikipedia.org/wiki/Procrustes_analysis) in Matlab. I'm sure it has a python version.
2. how did you actually fed your point cloud in fitting_pointclouds.py? use the point cloud aligned with the FLAME coordinate as the authors suggested that you just have to scale with a factor 4 from FLAME to NPHM coordinate system.

[Zvyozdo4ka]

@WXizhi Many thanks for your comments!

2. how did you actually fed your point cloud in fitting_pointclouds.py? use the point cloud aligned with the FLAME coordinate as the authors suggested that you just have to scale with a factor 4 from FLAME to NPHM coordinate system.

1. Where do you define scale factor?

2. May i ask you to share the installed packages in your environment? did you have problems with pytorch3d? Even though i am installing everything in accordance with instructions, i still have this issue:

 from pytorch3d.ops import knn_points, knn_gather
ModuleNotFoundError: No module named 'pytorch3d'

3. Did you get this dataset? https://docs.google.com/forms/d/e/1FAIpQLScG9BhoHelqV6GnT-z9P2TsGTJ2x_FPHxdnne_RmlRkbYPPQQ/alreadyresponded

Did you use pre trained dataset? https://drive.google.com/drive/folders/1dajUVhnYgRxbmX9CpAXDw702YYb0VHm9

I don't understand difference between nphm and npm data

[WXizhi]

@WXizhi Many thanks for your comments!

2. how did you actually fed your point cloud in fitting_pointclouds.py? use the point cloud aligned with the FLAME coordinate as the authors suggested that you just have to scale with a factor 4 from FLAME to NPHM coordinate system.

1. Where do you define scale factor?
2. May i ask you to share the installed packages in your environment? did you have problems with pytorch3d? Even though i am installing everything in accordance with instructions, i still have this issue:

 from pytorch3d.ops import knn_points, knn_gather
ModuleNotFoundError: No module named 'pytorch3d'

3. Did you get this dataset? https://docs.google.com/forms/d/e/1FAIpQLScG9BhoHelqV6GnT-z9P2TsGTJ2x_FPHxdnne_RmlRkbYPPQQ/alreadyresponded

Did you use pre trained dataset? https://drive.google.com/drive/folders/1dajUVhnYgRxbmX9CpAXDw702YYb0VHm9

I don't understand difference between nphm and npm data

1. Where do you define scale factor? as far as I can remember, the point cloud aligend to FLAME coordinate need to scale by 4 before feed into fitting_pointclouds.py. you can check it by overlapping custom data with nphm dataset.
2. pytorch3d issues I solved pytorch3d installation issues by : a. build with source. b. conform with the cuda version. just follow the instructions to build with source, it will be fine.
3. Did you get this dataset? Yes I did. thanks to the authors.
4. Do you use pre trained dataset? yes. I just tried to inference the mesh for facescape data. but did not dig deep in the repository.
5. difference between NPM and NPHM as far as I can remember, NPM is a previous and baseline version of NPHM.

[Zvyozdo4ka]

While the difference is subtle (you can notice it, especially for the chin and forehead), I guess it confuses the model quite a bit. Since the registration is quite consistent in the dataset this is a definitely a small outlier.

One suggestion would be to fit FLAME to the model first, since it shares the same coordinate system with NPHM (you just have to scale with a factor 4 from FLAME to NPHM coordinate system).

Why do we need scale with a factor 4 if flame and point cloud aligned properly? the output without scaling lies in same coordinate system

[Zvyozdo4ka]

@WXizhi, or @SimonGiebenhain What did you use to get FLAME model?